First implementation of dynare python library.

[albop]

Some elements of context:

• dynare preprocessor is needed to import model files describing economic models
• goal is to provide a way to solve / interact with models form within the browser.

After some lengthy discussion with @DerThorsten, I managed to:

• compile and run an executable version of dynare with wasm
• create a python shared library version with pybind11. This requires a forked version of dynare preprocessor, which is the source for may package.
• create a linux conda package which is essentially the same as the recipe from this PR recipe without the "cross-file" option in meson setup (in build.sh)

I don't know which steps I should take now and am waiting for some feedback from you (and from CI).

[albop]

Here is the error :
’’’
RuntimeError: Solver could not find solution.Mamba failed to solve:

• libboost-headers
• pthread-stubs
• pybind11
• emscripten-abi 3.1.45.*

with channels:

The reported errors are:

• Encountered problems while solving:
• • nothing provides requested libboost-headers
• • nothing provides requested pthread-stubs
• ’’’

[albop]

Now the errors during the compilation are:

• ../src/ModelTree.hh:413:17: error: unknown type name 'jthread'; did you mean 'thread'?
• ../src/ModelTree.cc:48:8: error: unknown type name 'jthread'; did you mean 'thread'?
• ../src/ModelTree.cc:1910:45: error: unknown type name 'stop_token'
• /home/pablo/emsdk/upstream/emscripten/cache/sysroot/include/c++/v1/thread:282:5: error: attempt to use a deleted function

and some repetition of the same errors.

These seem related to C++ standard library (thread, stop_token are defined in C++20). The preprocessor is usually compiled with -std=gnu++20.

[albop]

I have now removed all parts from the source which involve std:jstream.
Now compilation fails during the linking step with error: "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."

[DerThorsten]

the recipe it self uses a lot of build dependencies, but a lot of them should be host dependencies.
also in the build I see

$BUILD_PREFIX/include/python3.12/pyport.h:586:2: error: "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."

so its using a python from the build prefix which is wrong (the python from the build prefix is not for the wasm platform but for linux (or whatever machine we are on))

[DerThorsten]

smth like rm -rf $PREFIX/bin/python* in the build.sh before starting your build will fix the last error.
this is because there is only an libpython and no actual python binary for wasm. We replaced the binary with some script saying that there is no python binary, just removing that might do the trick

[albop]

Hmm, that is very clear. Now the situation starts to get more complicated as I need to find a way to have meson build a python extension_module without the python binary, or with a dummy one.

[DerThorsten]

often the cross_python package does just the right thing. It uses the "normal" python but sets everything up st. the build will be for emscripten-wasm32 (most python packages for emscripten-forge use this package)

[albop]

Sounds perfect. This conda world never ceases to amaze me! Do you have any example of a package that uses it by any chance?

[DerThorsten]

https://github.com/emscripten-forge/recipes/blob/main/recipes/recipes_emscripten/regex/recipe.yaml#L19 adding that is sometimes enough

[DerThorsten]

So, now that the build is passing, you probably want to know if the build is actually working. To do that, just add a file test_<WHATEVER_NAME>.py to the recipe.
Like here.

If this test is able to import your package we are done and can merge this =)

[DerThorsten]

is the extension compiled with the SIDE_MOUDLE=1 flag?

[albop]

I didn´t add it explicitly. I've tried locally to add -s SIDE_MODULE=1 to the linker but had no effect. However putting -s SIDE_MODULE=0 results in another error during the test (some std::bad_alloc).
Which one should I use ?

[DerThorsten]

I didn´t add it explicitly. I've tried locally to add -s SIDE_MODULE=1 to the linker but had no effect. However putting -s SIDE_MODULE=0 results in another error during the test (some std::bad_alloc). Which one should I use ?

since its a shared library it must be SIDE_MODULE=1

[albop]

Ok, I've added -s SIDE_MODULE=1 to the compiler options, but I suppose it was already the default with meson python extension.

[albop]

Ok, now I don't know. The error on CI: RuntimeError: [Error('LinkError: WebAssembly.instantiate(): Import #404 module="env" function="memory": mismatch in shared state of memory, declared = 1, imported = 0'), TimeoutError('Timeout 240000ms exceeded.')] led me to experiment with threads, including:

• compiling with -pthread via a dependence on threads in meson
• adding flag -sUSE_PTHREADS=1

But it didn't work.

Strangely the error I get on my laptop is different:

Test  backend : browser-main FAILED Error: need the dylink section to be first                                                                                                                                                                                
    at failIf (http://127.0.0.1:36925/pyjs_runtime_browser.js:8:42054)                                                                                                                                                                                        
    at Object.getDylinkMetadata (http://127.0.0.1:36925/pyjs_runtime_browser.js:8:42601)                                                                                                                                                                      
    at http://127.0.0.1:36925/pyjs_runtime_browser.js:8:13628                                                                                                                                                                                                 
    at Array.forEach (<anonymous>)                                                                                                                                                                                                                            
    at calculateGlobalLibs (http://127.0.0.1:36925/pyjs_runtime_browser.js:8:13567)                                                                                                                                                                           
    at Object.loadDynlibsFromPackage [as _loadDynlibsFromPackage] (http://127.0.0.1:36925/pyjs_runtime_browser.js:8:12076)                                                                                                                                    
    at Module.bootstrap_from_empack_packed_environment (http://127.0.0.1:36925/pyjs_runtime_browser.js:8:21504)                                                                                                                                               
    at async make_pyjs (eval at evaluate (:226:30), <anonymous>:25:4)                                                                                                                                                                                         
    at async eval (eval at evaluate (:226:30), <anonymous>:111:32)                                                                                                                                                                                            
    at async <anonymous>:252:30                                          

[albop]

@DerThorsten : do you have any idea of where to go next on this ?

[DerThorsten]

@albop its finally working!
is dynare-preprocessor-pylib the intended name?

[albop]

Absolutely amazing ! Thank you so much. Can you tell me in just a few words what did the trick ?
Yes, dynare-preprocessor-pylib is the best name for now. I haven't registered on conda-forge yet. Is that a problem ? If so I can start the process.
There is a conda-forge package named dynare-preprocessor but the idea is that it will eventually belong to upstream, so as long as the pylib work isn´t merged upstream in one form or another, I'd rather keep the two packages separates.

[DerThorsten]

removing all -pthread arguments, adding -fexception and -s WASM_BIGINT everywhere

[albop]

sounds easy when you say it like that...
Is there a way for me to experiment with the result or should I just wait until it appears in emscripten-forge ?

[DerThorsten]

sounds easy when you say it like that... Is there a way for me to experiment with the result or should I just wait until it appears in emscripten-forge ?

the package is available at https://prefix.dev/channels/emscripten-forge-dev/packages/dynare-preprocessor-pylib

You should be able to make lite-deployments with it using https://github.com/jupyterlite/xeus-lite-demo
note that you need to use the channel https://repo.prefix.dev/emscripten-forge-dev instead of https://repo.mamba.pm/emscripten-forge