Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Vegan for webR / wasm #623

Closed
isbool opened this issue Mar 4, 2024 · 26 comments
Closed

Vegan for webR / wasm #623

isbool opened this issue Mar 4, 2024 · 26 comments

Comments

@isbool
Copy link

isbool commented Mar 4, 2024

Hey,
Would it be possible for vegan to be compiled to emscripten to be able to be used with webR / wasm.
@jarioksa
Copy link
Contributor

jarioksa commented Mar 4, 2024

No idea. Why don't you try?

@gavinsimpson
Copy link
Contributor

gavinsimpson commented Mar 4, 2024 •
edited
Loading

I'd be very surprised if the reason that vegan isn't already compiled is because someone forgot to compile it. I suspect some mix of the Fortran or C code in the package is currently not something that is easily amenable to the emscripten compilation process.

@jarioksa
Copy link
Contributor

jarioksa commented Mar 5, 2024 •
edited
Loading

vegan is not available in the automatic builds from CRAN in https://repo.r-wasm.org. Obviously it fails. We can speculate for the reason, but that's futile: we should know why it failed to build.

However, here some speculation: we have three kind of call interface to compiled code:

  1. Most common (18 cases) is .Call with direct handling of R objects both as input and in return values.
  2. We have five .C interfaces which handle simple vector objects and actually knows nothing about R objects.
  3. We have two .Fortran calls. One of these is expandable: it is only used in orderingKM and no other vegan object needs that. The second one is monoMDS which is important for many vegan dependencies.

webR/wasm claims to be able to handle C, C++ and Fortran and from that point of view these should work. R assumes all these interfaces look similar to the calling function, and base R uses all of these. It is unclear to me if webR/wasm needs something special to handle all these.

Please note that decorana is based on Fortran code, but we do call it via .Call interface, and that C code will access Fortran subroutines using F77_CALL macro of R. However, if the Fortran call is the problem, but decorana works, we can have similar wrapping to .Call or .C interface in these functions. If calling Fortran from C is the problem, somebody should do a massive re-writing of a large part of the vegan source code. Surely, I won't do it.

More useless (in lack of logs) speculation: the wasm build seems to use Makevars like we do, and this may cause conflicts.

That was all background that can only become useful after we get the failure logs and see where is the problem. Without logs there is nothing we can do. Anybody can produce or access such logs?

@isbool
Copy link
Author

isbool commented Mar 5, 2024

I dont think i have the technical knowledge to try to build it.
if no one else can try at least can some one guide me on how to do that abit

@isbool
Copy link
Author

isbool commented Mar 6, 2024

what's interesting is that vegan is a dependency of GUniFrac, but GUniFrac is available for webR does that mean that they have compiled vegan ?

@jarioksa
Copy link
Contributor

jarioksa commented Mar 6, 2024 •
edited
Loading

@isbool GUniFrac is available, but they say that not all dependences are available, and vegan is listed as missing dependence. I don't know what are the criteria for this decision. The repo makes available a huge number of packages with missing dependencies: when writing this, the wasm repo had 19867 packages, but only 11773 had all their dependencies satisfied (CRAN had 20526 packages when writing this).

@isbool
Copy link
Author

isbool commented Mar 6, 2024 •
edited
Loading

@jarioksa i tried to compile it (using this proccess with docker) i get this:

> build("vegan")
v Updated metadata database: 6.19 MB in 4 files.
v Updating metadata database ... done
trying URL 'https://packagemanager.posit.co/cran/latest/src/contrib/vegan_2.6-4.tar.gz'
Content type 'binary/octet-stream' length 1490861 bytes (1.4 MB)
==================================================
downloaded 1.4 MB


> Will install 1 package.
> Will download 2 packages with unknown size.
+ permute   0.9-7 [dl]

i Getting 1 pkg with unknown size
v Got permute 0.9-7 (x86_64-pc-linux-gnu-ubuntu-22.04) (219.60 kB)
v Downloaded 1 package (219.60 kB) in 2s
v Installed permute 0.9-7  (18ms)
v 7 deps: added 1, dld 1 (219.60 kB) [2.7s]
* installing *source* package 'vegan' ...
** package 'vegan' successfully unpacked and MD5 sums checked
** using non-staged installation
** libs
using C compiler: 'emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 3.1.47 (431685f05c67f0424c11473cc16798b9587bb536)'
using Fortran compiler: 'flang-new version 17.0.6 (https://github.com/r-wasm/llvm-project 49f8a443424a6111bc055f349c9bfad0aa619c22)'
emcc -DNDEBUG   -I/opt/webr/wasm/include -I/opt/webr/R/build/R-4.3.3/build/include -I/opt/webr/R/build/R-4.3.3/src/include -s USE_BZIP2=1 -s USE_ZLIB=1  -fpic  -Oz -fPIC -fwasm-exceptions -s SUPPORT_LONGJMP=wasm -std=gnu11 -c data2hill.c -o data2hill.o
/opt/flang/host/bin/flang-new  -fPIC  -O2 -c decorana.f -o decorana.o
emcc -DNDEBUG   -I/opt/webr/wasm/include -I/opt/webr/R/build/R-4.3.3/build/include -I/opt/webr/R/build/R-4.3.3/src/include -s USE_BZIP2=1 -s USE_ZLIB=1  -fpic  -Oz -fPIC -fwasm-exceptions -s SUPPORT_LONGJMP=wasm -std=gnu11 -c getF.c -o getF.o
emcc -DNDEBUG   -I/opt/webr/wasm/include -I/opt/webr/R/build/R-4.3.3/build/include -I/opt/webr/R/build/R-4.3.3/src/include -s USE_BZIP2=1 -s USE_ZLIB=1  -fpic  -Oz -fPIC -fwasm-exceptions -s SUPPORT_LONGJMP=wasm -std=gnu11 -c goffactor.c -o goffactor.o
emcc -DNDEBUG   -I/opt/webr/wasm/include -I/opt/webr/R/build/R-4.3.3/build/include -I/opt/webr/R/build/R-4.3.3/src/include -s USE_BZIP2=1 -s USE_ZLIB=1  -fpic  -Oz -fPIC -fwasm-exceptions -s SUPPORT_LONGJMP=wasm -std=gnu11 -c init.c -o init.o
/opt/flang/host/bin/flang-new  -fPIC  -O2 -c monoMDS.f -o monoMDS.o
emcc -DNDEBUG   -I/opt/webr/wasm/include -I/opt/webr/R/build/R-4.3.3/build/include -I/opt/webr/R/build/R-4.3.3/src/include -s USE_BZIP2=1 -s USE_ZLIB=1  -fpic  -Oz -fPIC -fwasm-exceptions -s SUPPORT_LONGJMP=wasm -std=gnu11 -c nestedness.c -o nestedness.o
/opt/flang/host/bin/flang-new  -fPIC  -O2 -c ordering.f -o ordering.o
emcc -DNDEBUG   -I/opt/webr/wasm/include -I/opt/webr/R/build/R-4.3.3/build/include -I/opt/webr/R/build/R-4.3.3/src/include -s USE_BZIP2=1 -s USE_ZLIB=1  -fpic  -Oz -fPIC -fwasm-exceptions -s SUPPORT_LONGJMP=wasm -std=gnu11 -c pnpoly.c -o pnpoly.o
emcc -DNDEBUG   -I/opt/webr/wasm/include -I/opt/webr/R/build/R-4.3.3/build/include -I/opt/webr/R/build/R-4.3.3/src/include -s USE_BZIP2=1 -s USE_ZLIB=1  -fpic  -Oz -fPIC -fwasm-exceptions -s SUPPORT_LONGJMP=wasm -std=gnu11 -c stepacross.c -o stepacross.o
emcc -DNDEBUG   -I/opt/webr/wasm/include -I/opt/webr/R/build/R-4.3.3/build/include -I/opt/webr/R/build/R-4.3.3/src/include -s USE_BZIP2=1 -s USE_ZLIB=1  -fpic  -Oz -fPIC -fwasm-exceptions -s SUPPORT_LONGJMP=wasm -std=gnu11 -c vegdist.c -o vegdist.o
emcc -s SIDE_MODULE=1 -s WASM_BIGINT -s ASSERTIONS=1 -L/opt/webr/wasm/lib -L/opt/webr/wasm/R-4.3.3/lib/R/lib -s USE_BZIP2=1 -s USE_ZLIB=1 -fwasm-exceptions -s SUPPORT_LONGJMP=wasm -Oz -o vegan.so data2hill.o decorana.o getF.o goffactor.o init.o monoMDS.o nestedness.o ordering.o pnpoly.o stepacross.o vegdist.o -L/lib/R/lib -lRlapack -L/lib/R/lib -lRblas /opt/flang/wasm/lib/libFortranRuntime.a
wasm-ld: error: function signature mismatch: malloc
>>> defined as (i64) -> i32 in monoMDS.o
>>> defined as (i32) -> i32 in /opt/flang/wasm/lib/libFortranRuntime.a(ISO_Fortran_binding.o)
emcc: error: '/opt/emsdk/upstream/bin/wasm-ld -o vegan.so --whole-archive -L/opt/webr/wasm/lib -L/opt/webr/wasm/R-4.3.3/lib/R/lib data2hill.o decorana.o getF.o goffactor.o init.o monoMDS.o nestedness.o ordering.o pnpoly.o stepacross.o vegdist.o -L/lib/R/lib /opt/webr/wasm/R-4.3.3/lib/R/lib/libRlapack.so -L/lib/R/lib /opt/webr/wasm/R-4.3.3/lib/R/lib/libRblas.so /opt/flang/wasm/lib/libFortranRuntime.a -L/opt/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/pic --no-whole-archive -mllvm -combiner-global-alias-analysis=false -mllvm -wasm-enable-sjlj -mllvm -disable-lsr -mllvm -wasm-enable-eh -mllvm -exception-model=wasm --import-memory --strip-debug --export-dynamic --export-if-defined=main --export-if-defined=__get_exception_message --export-if-defined=free --export-if-defined=__cpp_exception --export-if-defined=__cxa_increment_exception_refcount --export-if-defined=__cxa_decrement_exception_refcount --export-if-defined=__thrown_object_from_unwind_exception --export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm --export-if-defined=__start_em_lib_deps --export-if-defined=__stop_em_lib_deps --export-if-defined=__start_em_js --export-if-defined=__stop_em_js --export-if-defined=__main_argc_argv --export-if-defined=__wasm_apply_data_relocs --export-if-defined=fflush --export=__wasm_call_ctors --experimental-pic -shared' failed (returned 1)
make: *** [/opt/R/4.3.2/lib/R/share/make/shlib.mk:10: vegan.so] Error 1
ERROR: compilation failed for package 'vegan'
* removing '/tmp/RtmpyRTq6H/file19f246e3/vegan'
Error in wasm_build(pkg, tarball_path, out_dir) :
  Building wasm binary for package 'vegan' failed.

@jarioksa
Copy link
Contributor

jarioksa commented Mar 6, 2024 •
edited
Loading

@isbool : thanks for the log. This seems to show the problem. However, I have no idea yet how to solve the problem.

The error occurs in monoMDS Fortran and there seems to be juggling and confusion between 64-bit integers (i64) and 32-bit integers (i32) in dynamically allocated storage. If I interpret the log correctly, we use 64-bit or i64 integers in Fortran, but the Fortran runtime specifies 32-bit or i32 integers. 64-bit integer is the R standard and also used in C-code. Compilation is OK, but linking fails. Need to inspect this more deeply. Perhaps somebody better in Fortran can lend a helping hand (@prminchin ?).

My first feeling is that we do nothing wrong, but the problem is in the 32-bit Fortran runtime in wasm instead of the 64 bit runtime. Compilation works, and monoMDS.o object file is produced, but it cannot be linked with the wasm Fortran runtime.

@isbool isbool mentioned this issue Mar 6, 2024
Open
@jarioksa
Copy link
Contributor

jarioksa commented Mar 6, 2024

There are other Fortran files in vegan which give no problem, but in monoMDS.f we allocate dynamically integer and double precision vectors. The log messages indicate (i64) -> i32 change in monoMDS.o, but then we get wasm-ld: error: function signature mismatch: malloc – so malloc may think differently of the types.

Here a grep of lines (with their numbers) dealing with dynamic allocation in the Fortran file monoMDS.f:

141:      INTEGER, ALLOCATABLE :: IWORK(:)
142:      DOUBLE PRECISION, ALLOCATABLE :: GRAD(:,:), GRLAST(:,:)

151:      ALLOCATE (IWORK(NDIS), GRAD(NOBJ,NDIM), GRLAST(NOBJ,NDIM))

359:      DEALLOCATE (IWORK, GRAD, GRLAST)

@prminchin
Copy link
Contributor

prminchin commented Mar 6, 2024 via email

@jarioksa
Copy link
Contributor

jarioksa commented Mar 6, 2024

@prminchin After more googling, I think it is a problem with the wasm build tools. It's possible that we could do something for this problem at the cost of ruining all other cases. This wasm is a Java application where you can run code in another computer in your own computer. Looking at this case, wasm people are worried that Fortran uses 64-bit integers instead of 32-bit integers, but we should be worried about using 32-bit integers. Let's see what kind of response we get, but certainly we will not downgrade code to accomodate wasm.

@prminchin
Copy link
Contributor

prminchin commented Mar 6, 2024 via email

@isbool
Copy link
Author

isbool commented Mar 6, 2024 •
edited
Loading

would it be possible for me to exclude this file and lose only some functionality, to be able to use the rest of vegan on wasm?
or its a core part of it ?
Also if as mention you decide that is unsafe to fix it, could it be possible that 2 versions exist or we have some instructions on how to compile it for wasm ?

@gavinsimpson
Copy link
Contributor

@isbool you can always maintain your own version of vegan, just fork the repo, remove what you want (and make sure it passes R CMD check as IIRC this is need for webr too). So you can delete whatever you want from your own fork. Then you can add your repo to r-universe and it will now build a wasm version for you. If you do this though, be sure to change at least the Maintainer field in DESCRIPTION (or the Authors@R declarations, which I think we use now) so that it points to you and your email and not Jari's, while maintaining all proper attributions and copyright notices.

That said, monoMDS is used in the functionality a lot of people use vegan for; metaMDS() for running non-metric MDS. So you'd be loosing one of the killer features.

@jarioksa
Copy link
Contributor

jarioksa commented Mar 6, 2024 •
edited
Loading

Like @gavinsimpson wrote, metaMDS depends on monoMDS, and many packages depending on vegan particularly depend on metaMDS. Moreover, there are several functions in vegan that build upon or extend or support metaMDS/monoMDS and you must remove all these. You can remove monoMDS, but you need much more to pass tests and build vegan.

Here some things you need to do (just out of memory without looking at the package):

  • remove src/monoMDS.f and edit src/init.c for the removal.
  • remove metaMDS.R and its component functions.
  • remove support functions for metaMDS (plot, print etc.) or *.metaMDS.R.
  • remove corresponding help files – man/metaMDS.Rd may be all you need to remove.
  • remove all these entries from the NAMESPACE file.
  • run R tests and fix things you need to fix – you'll find functions that depend on metaMDS that you should remove and also fix tests for metaMDS/monoMDS.

@isbool
Copy link
Author

isbool commented Mar 8, 2024 •
edited
Loading

I will attempt to do that, but its seems monoMDS and metaMDS are rooted deeply into vegan.
UPDATE I looked around and and metaMDS is used in adonis which i need.

Is there any chance that you might look into how possible is to actually figure out the issue exactly and the options to solve it ?

Also could i somehow force the compiler to compile to i32 integers to match wasm ?

Is there any chance building webR with Dragonegg as mention here can have help some how fix this?
image

@isbool isbool mentioned this issue Mar 8, 2024
Closed
@georgestagg
Copy link

This is a known bug in our fork of the LLVM flang-new compiler that we use as part of our WebAssembly toolchain for webR. Officially, LLVM Flang does not yet support compiling to WebAssembly, which is why we maintain a fork.

@jarioksa is right; the issue is that dynamic allocation with ALLOCATE() does not currently work with our Fortran compiler. The Fortran runtime library tries to call malloc() with an incorrectly typed argument for the 32-bit Wasm environment. The issue arises because we are cross-compiling from a 64-bit host machine to the wasm32 target.

I'm investigating, but I don't yet have a strong idea of when a fix will be available.

This is not the only R package affected. It is highly likely there is nothing wrong with the code in this package, especially if it compiles fine on other systems.

I would recommend waiting until support for ALLOCATE() is fixed in our compiler. An option, if possible, would be to rework the Fortran code not to use dynamic allocation. But as I say, this is really a webR problem that I hope to be fixed in the near future.

@jarioksa
Copy link
Contributor

jarioksa commented Mar 8, 2024 •
edited
Loading

@georgestagg @prminchin @isbool – Before inspecting the code (what a blessing!) I think it may be possible to change the code so that it does not use dynamic allocation (ALLOCATE in Fortran, seemingly cascading to malloc in C). In principle – in practice it may be hard work. The Fortran code is derived from a main program where dynamic allocation is necessary, but we are using it as a subroutine which means that we could reserve the storage in the calling routine and pass the arrays from the calling program. This is not something that a Real Programmer would like to do for internal work arrays that are not needed in the calling program, but it could solve the issue while the LLVM flang is not yet fixed.

@georgestagg : we got this error message for an INTEGER array, but we also have two DOUBLE PRECISION (like it used to be called way back when I wrote Fortran) arrays. We did not get any errors from those, but is this only because the build stopped with the first error?

@jarioksa
Copy link
Contributor

jarioksa commented Mar 8, 2024

@isbool: @prminchin sent me a revised version of monoMDS Fortran code without dynamic allocation. I'll check how to merge that in vegan (it also needs changes in the R side). However, a new version should be available in github over the weekend.

@isbool
Copy link
Author

isbool commented Mar 10, 2024

Hey, how its going? any unexpected hiccups ?

@gavinsimpson
Copy link
Contributor

With all due respect; it's the weekend, and no one here is paid to work on vegan. Sit tight and I'm sure someone will update the thread if progress is made.

@isbool
Copy link
Author

isbool commented Mar 10, 2024

yea yea no pressure i just asked, if there was something i could do to help, no pressure

jarioksa pushed a commit that referenced this issue Mar 13, 2024
@prminchin @jarioksa
remove dynamic allocation: fixes issue #623
75e347d 
Vegan could not be built for webR/wasm because their Fortran
compiler (under development) did not handle dynamic allocation.

This needed some hand-editing as the fix was not based on the
latest github version and contained removed subroutines.
@jarioksa
Copy link
Contributor

Commit 75e347d removes dynamic allocation, and if that was the only problem, vegan should work with webR / wasm.

@jarioksa
Copy link
Contributor

Can anybody confirm that vegan can be built in webR / wasm?

I don't have tools to test this.

@georgestagg
Copy link

I can confirm that vegan now builds for webR. Here is a a screenshot of a local build:

Screenshot 2024-03-21 at 13 10 53

I'm currently working on the release of webR 0.3.0. Once that is done, the webR binary package repository will be unfrozen, all packages will be upgraded, and a working vegan will be available for general use.

@jarioksa
Copy link
Contributor

Thanks! Closing this as completed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@gavinsimpson @georgestagg @jarioksa @prminchin @isbool