New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Emterpreter and changing function names #7988

Open
Beuc opened this Issue Feb 3, 2019 · 4 comments

Comments

Projects
None yet
2 participants
@Beuc
Copy link
Contributor

Beuc commented Feb 3, 2019

(Note: I tried sending this twice to the emscripten-discuss mailing list, and it was silently dropped each time... Not sure what it didn't like)

When compiling my RenPyWeb project, which use Emterpreter, I often get
errors about new, non-whitelisted functions such as:

___Pyx_PyObject_CallNoArg_387
___Pyx_PyObject_CallNoArg_6345

This may happen when I make some changes in the code or in the
compilation flags.

AFAIU these are generated by Emscripten ('grep' only sees references in
Emscripten-generated files) and the numbers are some kind of offset.

This is annoying because I usually have to compile (5-10mn depending on
target), run in a browser and wait for complete startup (couple
minutes), get the new stack strace, edit my Makefile and
EMTERPRETIFY_WHITELIST, and repeat.
And I need to tell my users/contributors to do the same

Is there anything I can do to avoid these dynamic function names?

@kripken

This comment has been minimized.

Copy link
Member

kripken commented Feb 4, 2019

I think the source of those numbers is the linker - it adds suffixes when names would otherwise collide (e.g. two files might contain static ___Pyx_PyObject_CallNoArg so it has to differentiate them when linked together). I'm not sure how that works, but I guess it's not deterministic in a predictable way?

One option here may be to support wildcards in the whitelist, perhaps. Another is you can use llvm-nm on the final linked bitcode, to see what names it emitted, and can have a script in your build syste to generate the whitelist from that.

@Beuc

This comment has been minimized.

Copy link
Contributor Author

Beuc commented Feb 4, 2019

Those are static conflicts indeed:

gen/pygame_sdl2.transform.c:static CYTHON_INLINE PyObject* __Pyx_PyObject_CallNoArg(PyObject *func);
$ for i in *.o; do echo -n $i:; llvm-nm $i|grep CallNoArg || echo; done
pygame_sdl2.color.o:-------- t __Pyx_PyObject_CallNoArg
pygame_sdl2.controller.o:-------- t __Pyx_PyObject_CallNoArg
...
pygame_sdl2.surface.o:-------- t __Pyx_PyObject_CallNoArg
pygame_sdl2.transform.o:-------- t __Pyx_PyObject_CallNoArg

I don't know how the number is computed. It's moderately stable (doesn't change everytime), it's not the line number, and it changes depending on other files linked and possibly the linker options used.

With a 2-steps compilation (*.bc -> monolith.bc -> .html), I can see some matching symbols (modulo "." -> "_"):

-------- t __Pyx_PyObject_CallNoArg
-------- t __Pyx_PyObject_CallNoArg.10863
-------- t __Pyx_PyObject_CallNoArg.1301
-------- t __Pyx_PyObject_CallNoArg.1847
-------- t __Pyx_PyObject_CallNoArg.2057
-------- t __Pyx_PyObject_CallNoArg.216

which I can crudely flag:

-s EMTERPRETIFY_WHITELIST='["_main", ... $(shell llvm-nm build/t/index.bc | grep -Po '__Pyx_PyObject_CallNoArg\.\d+' | sed -e 's/\./_/' -e 's/.*/"&"/' | tr '\n' ',') ...]'

(I had to LOL though ;))

One issue is I cannot distinguish which ones to whitelist (though in this particular case these functions are reasonably quick).
If we need to live with this imprecision, a wildcard makes more sense.

@kripken

This comment has been minimized.

Copy link
Member

kripken commented Feb 4, 2019

Heh, nice shell command :)

Yeah, +1 for a whitelist wildcard then.

@Beuc

This comment has been minimized.

Copy link
Contributor Author

Beuc commented Feb 10, 2019

For reference my work-around is flawed - and I don't understand how it worked until I upgraded today...
Missing a leading '_', and not executed by 'make' at the right time.

-s EMTERPRETIFY_WHITELIST='["...", '`llvm-nm build/t/index.bc | grep -Po '__Pyx_PyObject_CallNoArg\.\d+' | sed -e 's/\./_/' -e 's/.*/"_&"/' | tr '\n' ','`' "..."]'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment