Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

embind: Fix method pointers for AOT JS generation. #21168

Merged
merged 1 commit into from Jan 26, 2024
Merged

embind: Fix method pointers for AOT JS generation. #21168

merged 1 commit into from Jan 26, 2024

Conversation

brendandahl
Copy link
Collaborator

Previously, AOT JS glue code for method pointers used the address of the method pointer as a unique ID to match JS glue with the binding. However, sometimes static initializers are run in different orders when running the AOT generation code versus when the code is run normally. This caused the method pointer address to be different and the binding didn't match up.

Instead of using the function pointer or the method pointer address we now use a generated signature for the invoker to match up the glue code. This works for both types of pointers and also has the advantage that many of the glue code functions are nearly identical and can be reusued (closure arguments make them behave differently).

Fixes #20994

@brendandahl
Copy link
Collaborator Author

My little benchmark for code size improves quite a bit, but it is probably not a good test anymore since it had lots of repeated function signatures.

Before (bytes):
hun_aot_js.js : 37,673
hun_aot_js.js.gz: 7,475

After (bytes):
hun_aot_js.js : 15,044
hun_aot_js.js.gz: 6,310

sbc100
sbc100 approved these changes Jan 25, 2024
View reviewed changes
test/other/embind_jsgen_static_init.cpp Outdated Show resolved Hide resolved
src/embind/embind_gen.js Outdated Show resolved Hide resolved
src/embind/embind_shared.js Outdated Show resolved Hide resolved
src/embind/embind_gen.js Outdated Show resolved Hide resolved
@brendandahl
embind: Fix method pointers for AOT JS generation.
1daa248 
Previously, AOT JS glue code for method pointers used the address of the
method pointer as a unique ID to match JS glue with the binding. However,
sometimes static initializers malloc different sizes when running
the AOT generation code versus when the code is run normally. This caused
the method pointer address to be different and the binding didn't match up.

Instead of using the function pointer or the method pointer address we now
use a generated signature for the invoker to match up the glue code. This
works for both types of pointers and also has the advantage that many
of the glue code functions are nearly identical and can be reusued (closure
arguments make them behave differently).

Fixes emscripten-core#20994
@brendandahl brendandahl merged commit 6ae900e into emscripten-core:main Jan 26, 2024
25 of 27 checks passed
lewing pushed a commit to dotnet/emscripten that referenced this pull request Apr 12, 2024
@radekdoulik @brendandahl
@sbc100 @dependabot @ifiddynine @kainino0x @kripken @kleisauke @xdestiny110 @ypujante @waywardmonkeys @mjgrzymek @coopersimon @mrolig5267319 @kangtastic @dschuff @wiertek @curiousdannii @past-due @FrancoisRecisio @wojdyr @hoodmane @cwoffenden @Daft-Freak @ericoporto @aheejin @patrickkettner @SingleAccretion @jnickg @shaoboyan @nsfisis
Merge upstream 3.1.56 (#64)
baa07c9 
* embind: Ensure embind works under c++11. (emscripten-core#21100)

We no longer need to test c++17 since that is the default used, but we
should ensure we maintain compatibility with c++11.

* Fix C++ function signatures in mimalloc (emscripten-core#21127)

The C++ functions that take `std::nothrow_t` all take a reference.
Without this change I'm seeing function signature mismatches because the
final argument is not pointer-width.  I'm guessing this simply went
unnoticed because most platforms don't do signature checking in the
linker and this just happens to work.

e.g.  `_ZnwjRKSt9nothrow_t` which demangles to `operator new(unsigned
int, std::nothrow_t const&)`

See upstream bug at microsoft/mimalloc#840.

* Bump github/codeql-action from 2.22.5 to 3.23.1 (emscripten-core#21134)

Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2.22.5 to 3.23.1.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](github/codeql-action@74483a3...0b21cf2)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [test] Pass filename explicitly to `compile_btest` helper. NFC (emscripten-core#21103)

This allows us to use `compiler_for` to determine whether to run
`EMCC` or `EMXX` appropriately.

* Remove some unnecessary complexity in `create_receiving`. NFC (emscripten-core#21131)

* Fix makeBlendState assert for non-existent nextInChain (emscripten-core#21119)

* Use single quotes in JS code where possible. NFC (emscripten-core#21133)

For some reason this actually saves on code size too.

* WebGPU: Fill out webgpu-externs.js (emscripten-core#21144)

Fixes test browser.test_webgpu_basic_rendering_closure by fixing
wgpuShaderModuleGetCompilationInfo (compilationInfo.messages.length).

Fixes: emscripten-core#20415

* Remove second argument to `ErrnoError`. NFC (emscripten-core#21136)

There was only one single place (out of hundreds) where this was set
and one single place in `doStat` where it was read.

I removed the `doStat` workaround by updating `mayLookup` instead.

* [Docs] Mention information that may exist in unoptimized builds (emscripten-core#21147)

Fixes emscripten-core#21084

* Convert test_stat_fail_alongtheway to a file based test (emscripten-core#21138)

Also remove the special logging and just use assert which fails
fast and gives a backtrace.

* Minor webidl binder fixes (emscripten-core#21152)

Split out from emscripten-core#21151.

The only semantic change here is the fix the type of second argument to
to the array member getter function (previously it was using
`m.type` which is the array type itself, but that second argument to the
setter function is actually the inner type `m.type.inner`.

This enables the type of this argument to be checked correctly, which in
turn required a minor fix to the test case.  This only effects
`IDL_CHECKS=all` builds.

* [ci] Pin firefix to dev-edition until the threading issue is resolved. NFC (emscripten-core#21164)

See emscripten-core#21163

* Bump expected LLVM version from 18 to 19. NFC (emscripten-core#21165)

* Add LLVM version bump to ChangeLog. NFC (emscripten-core#21166)

* Remove use of `demangleAll` helper function (emscripten-core#21156)

This was used to demangle C++ symbols JS stacktraces.  However for
a long time now native symbols in JS backtraces are already demanged
because names in the name section are in demangled form.  So this
function has not done anything for a very long time.

As a followup I think we can probably completely remove the
`DEMANGLE_SUPPORT` settings.

* Rebaseline codesize expectations. NFC

* Rebaseline codesize expectations. NFC

* Use JavaScript class for FS.ErrnoError. NFC (emscripten-core#21149)

* [wasm64] Update webidl to support wasm64 (emscripten-core#21151)

* Rebaseline codesize expectations. NFC

* Acorn optimizer: Allow top-level await expressions (emscripten-core#21117)

`-pthread` + `-sEXPORT_ES6` emits a top-level await expression
in the main worker file after commit 49ab23a. Ensure the Acorn
optimizer is aware of this.

* fix wgpu get dimension bug (emscripten-core#21173)

Co-authored-by: Jerry.Zhuang <jerry.zhuang@jwzg.com>

* Avoid unnecessary dependency on `SYSCALLS` object. NFC (emscripten-core#21170)

I noticed that `proc_exit` was depending on `SYSCALLS` even though
it wasn't actually using `SYSCALLS`.

* embind: Fix method pointers for AOT JS generation. (emscripten-core#21168)

Previously, AOT JS glue code for method pointers used the address of the
method pointer as a unique ID to match JS glue with the binding. However,
sometimes static initializers malloc different sizes when running
the AOT generation code versus when the code is run normally. This caused
the method pointer address to be different and the binding didn't match up.

Instead of using the function pointer or the method pointer address we now
use a generated signature for the invoker to match up the glue code. This
works for both types of pointers and also has the advantage that many
of the glue code functions are nearly identical and can be reusued (closure
arguments make them behave differently).

Fixes emscripten-core#20994

* [wasm64] Error on `-fsanitize=address` + wasm64 (emscripten-core#21180)

This chance also includes a couple of minor fixes to asan + memory64
that helped me to confirm that it really doesn't work yet.

See emscripten-core#21177

* [wasm64] Enable freetype test. NFC (emscripten-core#21179)

The poppler test also seesm to work fine under wasm64, just not the >4Gb
mode.

* [ci] Split browser64_4gb into its own runner (emscripten-core#21182)

Also add test_TextDecoder to the list of test.

* [wasm64] Fix glSharedSource in >4gb mode and enable test under wasm64 (emscripten-core#21184)

* Add profiling to `read_ports`. NFC (emscripten-core#21194)

See emscripten-core#21191

* Remove unused arguments from test_emmalloc_trim. NFC (emscripten-core#21181)

This test was originally added with these unused args. Perhaps while
it was being developed in was parameterized, but is not anymore.

* Fix GL.getSource bug introduced in emscripten-core#21184 (emscripten-core#21197)

This only seems to show up in `browser.test_sdl_ogl_regal`.

* Move emscripten_webgl_init_context_attributes to native code. NFC (emscripten-core#21189)

* Rebaseline codesize expectations. NFC

* Allow-list 7 more WebGL extensions (emscripten-core#21185)

These 6 extensions don't add any entry points, so shouldn't require any
Emscripten support (aside from having any enums present in gl2ext.h,
which they do):

- EXT_conservative_depth
- NV_shader_noperspective_interpolation
- EXT_depth_clamp
- WEBGL_blend_func_extended
- WEBGL_compressed_texture_etc
- WEBGL_compressed_texture_etc1

Finally:

- WEBGL_compressed_texture_astc does add an entry point
getSupportedProfiles() in WebGL. This is a WEBGL_ extension, so it is
still safe to allow-list without implementing a C version of that
function. There is no corresponding GLES entry point for this; instead,
profile support is indicated by the presence of
KHR_texture_compression_astc_{ldr,hdr}. Emscripten should emulate those
extensions by checking getSupportedProfiles(), but I haven't implemented
this. Applications can still use the WEBGL_ version directly.

  WEBGL_compressed_texture_astc guarantees support for the LDR profile
  (since HDR is a superset of LDR, and there are only two profiles), so
  applications that only need LDR don't need to call this function. It
  can also be easily called using EM_ASM in applications that do use the
  HDR profile.

Fixes emscripten-core#21183

* Run unsafe_optimizations.js again after closure (emscripten-core#21196)

This allows `unsafe_optimizations.js` to cleanup some thing that
closure compiler was able to remove usages of.

Split out from emscripten-core#21195

* Mark test_itimer_proxy_to_pthread as flaky. NFC (emscripten-core#21175)

This test recently flaked on window too.

* Cleanup formating in audio_worklet JS code. NFC (emscripten-core#21203)

* Mark 3.1.53 as released (emscripten-core#21211)

* Don't force the use of imported memory under WASM2JS (emscripten-core#21195)

Setting `IMPORTED_MEMORY` can have other implications and don't want t
by default just because we chose WASM2JS.

The there is a very minor code size regression here that only effects
WASM2JS, but when combined with emscripten-core#21196 the overall effect is still a
win.

See emscripten-core#21071

* Use arrow function in library_webgl.js. NFC (emscripten-core#21186)

* Handle `-Wl,--stack-first` being passed explicitly (emscripten-core#21216)

Fixes: emscripten-core#21215

* [debug] Use two digits for the temporary file counter. NFC (emscripten-core#21207)

This means that filenames will sort correctly under normal sorting rules.

* [wasm64] Fix fetch api under wasm64 (emscripten-core#21219)

See emscripten-core#21177

* Rename/cleanup fetch tests. NFC (emscripten-core#21218)

* Make use of `emscripten_supports_offscreencanvas`. NFC (emscripten-core#21204)

* Use JS naming convention for JS-only symbol. NFC (emscripten-core#21188)

The `emscripten_` naming convention is something we use for native
function and this symbol is not visible to native code.

* Run more of out tests with `-sNO_DEFAULT_TO_CXX` (emscripten-core#21096)

We were previously only applying this flag to tests that use
`self.build` (i.e. most test).  Now we apply it to and even larger set.

The code size changes here only occur in debug builds where we enumerate
all known JS symbols, and there are come JS symbols are the C++ only.

* Some fixes for library_webgl.js in CAN_ADDRESS_GB mode (emscripten-core#21220)

This is an overall size win too after compression.

* When parsing settings as JSON, check the types we get back (emscripten-core#21206)

* Rebaseline codesize expectations. NFC

* More use of expectToReceiveOnModule in library_browser.js. NFC (emscripten-core#21224)

* Fix emscripten_webgl_make_context_current under wasm64/CAN_ADDRESS_2GB (emscripten-core#21228)

The first problem here was that
emscripten_webgl_make_context_current_calling_thread had the
wrong signature.

The second issue was use of `>> 2` on a pointer in
`emscripten_webgl_do_create_context` which doesn't work under wasm64
or CAN_ADDRESS_2GB.

* [embuilder] Add support for '*' wildcard for matching library names (emscripten-core#21236)

This is very useful for doing things like `./embuild build libGL*` to
`./embuilder clean libc*`.

* [test] Remove also_wasm2js btest argument. NFC (emscripten-core#21232)

This only have one user, and we have a cleaner mechanism for doing
this these days in terms of the decorator.

* Add --use-port option (emscripten-core#21214)

* [wasm64] Fix glUniformMatrix4fv (emscripten-core#21235)

* embind - Fix reading various bindings types from val on wasm64. (emscripten-core#21225)

- The function simpleReadValueFromPointer was always reading pointers as
i32 numbers which was incorrect on wasm64.
- Decoding of std::wstring was using a right shift to read pointers. Right
  shift was truncating numbers larger than i32 and causing misreads.
- Added tests to execute readValueFromPointer on all the binding types
  to ensure these don't break on wasm64.

* Move internal webgl signatures to a separate file. NFC (emscripten-core#21239)

This is in preparation for enabling these functions under wasm64.

* Avoid redundant function declarations in gl.c. NFC (emscripten-core#21237)

* Use @parmeterize in more browser tests. NFC (emscripten-core#21230)

* embind: Use optional return type for vector and maps. (emscripten-core#21102)

This helps create better TypeScript definitions for what
is actually returned.

* [test] Make more use of `also_with_wasm2js`. NFC (emscripten-core#21242)

* Support WGPUPrimitiveDepthClipControl / unclippedDepth (emscripten-core#21231)

Co-authored-by: Kai Ninomiya <kainino@chromium.org>

* [wasm64] Fix library_glemu.js under wasm64 (emscripten-core#21241)

* Fix test_cubegeom_pre_relocatable (emscripten-core#21240)

This test was broken by emscripten-core#21103 but when unnoticed because we run our
browser tests with swiftshader enabled, which skips this test.

* Consistent use of ValueError when parsing settings. NFC (emscripten-core#21245)

This is in preparation for a change to avoid using json parsing for
settings by default.

It also adds more context to the errors since when the ValueError
is caught higher up we print nice error messages.

* [wasm64] Fix wasm64 memory read in Fetch.js (emscripten-core#21246)

This bug only showed up under wasm64 when the address of the fetch
object was between 2Gb and 4Gb.  This causes the JS ">> 2" operation to
generate a negative number becuase the high bit is set:

```
$ node
> a = 2**31 + 10
2147483658
> a >> 2
-536870910
>
```

In `browser64_4gb` mode this bug resulted in a read from the first 4gb
of memory somewhere, which results a in 0 whereas read from a negative
address yields `undefined`.

* Fix broken compilation of regal tests (emscripten-core#21248)

This should have been part of emscripten-core#21096.  These failures don't show up in
CI because they are disabled under swiftshader.

* Minor formatting cleanup in src/library_glemu.js. NFC (emscripten-core#21249)

* Fix glDrawElements under CAN_ADDRESS_2GB (emscripten-core#21247)

* Re-enable preprocessing of pre/post JS file. NFC (emscripten-core#21227)

This change got reverted in emscripten-core#19006 so this time we make the
preprocessing optional.

This is useful as it allows things like `{{{ POINTER_SIZE }}}`
and `{{{ makeGetValue(..) }}}` to be used in pre/post JS files, just
like they can be in JS library files.

This change allows threadprofiler.js to be fixed such that it works
under wasm64.

Fixes: emscripten-core#21226

* Fix glDrawElements under CAN_ADDRESS_2GB (emscripten-core#21251)

This is partially a workround for emscripten-core#21250

* Fix capitalization of "GitHub". (emscripten-core#21257)

* docs: Remove discussion of using Travis CI. (emscripten-core#21256)

These are outdated and the docker images mentioned here haven't
been updated in 4 years.

* [test] Simplify post_manual_reftest. NFC (emscripten-core#21243)

Rather than embedding `reftest.js` directly in the html, simply include
it via script src attribute.

* Add contrib ports mechanism (emscripten-core#21244)

* Simplify get_subresource_location. NFC (emscripten-core#21260)

This function only ever gets called with no arguments.

I also confirmed that when we use a memory init file, that file always
exists.

* Improve manual_wasm_instantiate.html (emscripten-core#21258)

Modernize this test a little by using fetch API and JS spread operator.

* [Docs] Mention emscripten_exit_with_live_runtime wrt pthreads (emscripten-core#8167)

* Cleanup test_async_compile. NFC (emscripten-core#21259)

* Fix glUniformMatrix4fv in CAN_ADDRESS_2GB mode (emscripten-core#21264)

* Fix "emscripten emit" typo in Optimizing Code - WebAssembly section (emscripten-core#21269)

* Use JS string templates in src/library_glemu.js. NFC (emscripten-core#21261)

* Use rest and spread operators in multi-threaded code. NFC (emscripten-core#21270)

These operators are available everywhere that we can run mutli-threaded
code.

* [Docs] Added note to clarify new glfw3 port (emscripten-core#21272)

* Support multiple argument in pthread err/dbg wrappers (emscripten-core#21266)

* Cleanup generate_traditional_runtime_html. NFC (emscripten-core#21267)

- Use fetch() over XHR.
- No need for tryParseAsDataURI. This code is already in a `not
  settings.SINGLE_FILE` block which means URL for the wasm binary
  will never be data URI.  See `get_subresource_location`.

* WebGPU: Swapchain getCurrentTexture (emscripten-core#21271)

* WebGPU: added GetCurrentTexture method to swapchain.

* WebGPU: fix for GetCurrentTexture signature.

* Revert "[ci] Pin firefix to dev-edition until the threading issue is resolved. NFC (emscripten-core#21164)" (emscripten-core#21282)

This reverts commit 0eb6567.

The upstream has been fixed: https://bugzilla.mozilla.org/show_bug.cgi?id=1876355#c3

* [test] Use modern list format for `-s` setting. NFC (emscripten-core#21280)

* Fix `#preprocess` detection on windows (emscripten-core#21284)

Followup to emscripten-core#21227

* Update EMSCRIPTEN_WEBGL_CONTEXT_HANDLE to handle full pointer range (emscripten-core#21281)

In particular this changes the return type of
`emscripten_webgl_create_context` such that negative values can no
longer be returned.

This allows pointers (specifically 32-bit pointers over 2gb) to be
returned which is needed for pthread builds.

The only valid failure return code for `emscripten_webgl_create_context`
is now 0/NULL.  As it happens this function never fails with anything
except 0 so this should not be an impactful change.

Split out from emscripten-core#21268

Fixes: emscripten-core#21278

* [Docs] Document EXPORT_ES6 implicit behavior (emscripten-core#21274)

* It is implicitly enabled for .mjs output.
* MODULARIZE is implicitly enabled when EXPORT_ES6 is set.

* Remove redundant step in promise change. NFC (emscripten-core#21285)

This stop in the chain is only need if the offset converter is needed.

* Fix some typos [NFC] (emscripten-core#21275)

* Make use of JS rest and spread operators throughout the codebase. NFC (emscripten-core#21283)

Following up on emscripten-core#21270.  These operators were introduced way before
our current default set of browser versions.  Anyone targeting an
browser so old as to not support these will already be on the
transpilation path.

* [NFC] Assert we do not use --low-memory-unused when STACK_FIRST (emscripten-core#21291)

Assuming low memory was unused when the stack was first - which is where low
memory is - would be wrong. We never did that wrong thing as we only use that
optimization when not optimizing. Add an assertion + tests to prevent regressions.

* Cleanup some pthread tests. NFC (emscripten-core#21293)

* Refactor Emval implementation to avoid JS heap allocations. (emscripten-core#21205)

* Rebaseline codesize expectations. NFC

* Fix deadlock in test_pthread_run_on_main_thread (emscripten-core#21296)

Fixes: emscripten-core#18210

* [test] Remove some unneeded `-sINITIAL_MEMORY` settings. NFC (emscripten-core#21295)

I think the reason a lot of these are no longer needed is because
the default pthread stack size is less that it used to be.

* Fix broken JS glue for AUDIO_WORKLETS with EXPORT_ES6 (emscripten-core#21192)

WASM Audio Worklets with EXPORT_ES6 may break at runtime:

  test.js:989 Uncaught (in promise) TypeError: Cannot set property wasmTable of #<Object> which has only a getter
      at receiveInstance (test.js:989:25)
      at receiveInstantiationResult (test.js:1011:5)

The read-only getter at issue is created in ASSERTIONS-enabled
builds, and conflicts with the current way of exporting wasmTable
on the Module object.

Exporting wasmTable via EXPORTED_RUNTIME_METHODS prevents the getter
from being created in normal builds. In MINIMAL_RUNTIME builds, we
make sure to delete the getter before manually exporting as before.

We also prevent an ES6 Audio Worklet from loading the .wasm binary
via `new URL()`, as `URL` is unavailable in AudioWorkletGlobalScope.

* Disable tests that rely on changing output from size and llvm-objdump (emscripten-core#21298)

* Add options support to contrib ports (emscripten-core#21276)

* Fix MAIN_THREAD_EM_ASM_INT + CAN_ADDRESS_2GB (emscripten-core#21292)

We were using the sign bit of an integer to distinguish between data
pointers and fixed JS function indexes, but that doesn't work once
that data address can be larger than 2^31.

Technically this is very unlikely in practice since in order to get
an EM_ASM address over 2^31 you would either need 2Gb of static data
to be using `-sGLOBAL_BASE=2gb` like we do in the tests.

An alternative approach here would be assume we have fewer than
`GLOBAL_BASE` (1024 is most cases) proxied JS library functions and then
we could assume that small integers we JS library functions and larger
ones were data pointers (EM_ASM functions).  However that seems fragile
too.  Passing an extra argument around seems like a small cost here.

* WebGPU: Remove unused signalValue argument (emscripten-core#21299)

This was never implemented and was removed from Dawn and upstream webgpu.h a long time ago (and the JS API even longer ago).

(This will be a tiny breaking change for a few apps but we're not stable
and still need to do many more breaking changes.)

* Rebaseline codesize expectations. NFC

* Move __glGenObject to GL.genObject. NFC (emscripten-core#21302)

This function logically belongs alongside getNewId.

Split out from emscripten-core#18874

* Rebaseline codesize expectations. NFC

* [test] Fix test_pthread_run_on_main_thread under asan. NFC (emscripten-core#21301)

The output of this test is racey without this change and has been failing on the
emscripten-releases waterfall in the asan configuration.

We only just started caring about the precise output in emscripten-core#21296.

* [test] Remove some unnecessary usage of TOTAL_MEMORY and INITIAL_MEMORY. NFC (emscripten-core#21303)

* Fix remaining webgl + CAN_ADDRESS_2GB issues (emscripten-core#21306)

* Add --quiet option to file_packager (emscripten-core#21307)

This suppressed the `Remember to build the main file with ..` message
that is otherwise printed to stderr.

This also allows the tests to assert that no stderr is produced in
successful cases.

* [wasm64] Fix emscripten_get_preloaded_image_data (emscripten-core#21308)

Here we extract an inner JS function that take a JS string so that the
externally facing once can assume its being passed a pointer.

* Link to bug in test_webgl_multi_draw test. NFC (emscripten-core#21309)

* Add option to emit TypeScript definitions for Wasm module exports. (emscripten-core#21279)

The new flag `--emit-tsd <filename>` will generate a TypeScript defintion
file for any Wasm module exports. If embind is also used the definitions
for those types will also be included in the same file.

This still doesn't give the full picture of Wasm module e.g. missing
HEAP<N> and the various helper functions defined in JS.

* Fix wasm worker tests in high memory modes (emscripten-core#21319)

* Don't define JS versions of abort or __assert_fail in standalone mode (emscripten-core#21317)

In STANDALONE_WASM mode these functions are defined in libstandalonewasm
and not in JS.

The JS definitions meant they were not able to be added to
EXPORTED_FUNCTIONS.

Fixes: emscripten-core#21312

* embind- Fix tsgen failure from merge. (emscripten-core#21320)

Forgot to pull in the upstream branch before updating this test.

* Don't include `main.c` in libSDL_mixer (emscripten-core#21322)

This was broken when sdl2_mixer was updated recently in in emscripten-core#21013.

Fixes: emscripten-core#21321

* Run all browser tests in 2gb mode (emscripten-core#21268)

* Update emsymbolizer and emsize tests (emscripten-core#21305)

LLVM rev 8b0f47bfa updated section addresses to use binary offsets in linked
wasm files, which is reflected in the llvm-size (and thus emsize) tool. It
also made llvm-objdump display file offsets rather than section offsets when
disassembling binaries. Update the emsize and emsymbolizer tests accordingly
(expect section addresses in emsize, and use file rather than section offsets
in the emsymbolizer test, since the offsets are read from objdump output).

* [wasm64] Fix makeHEAPView for addresses over 4gb (emscripten-core#21324)

This enables all the cubegeom tests to pass

* Rebaseline codesize expectations. NFC

* Fix typo in dynamic linking debug prints (emscripten-core#21329)

* ChangeLog.md typo: INCOMING_MODULE_API should be INCOMING_MODULE_JS_API (emscripten-core#21314)

* embind - Disable wasm2js when emitting types. (emscripten-core#21333)

Fixes emscripten-core#21315

* Refactor `makeHEAPView` to use `getHeapOffset`. NFC (emscripten-core#21332)

The code for calculating the offsets was being duplicated here.

Also, optimize getHeapOffset to avoid any shifting when size is 1.

* Add suppport for external ports (emscripten-core#21316)

* [test] Improve test_sdl2_key and test_sdl2_mouse. NFC (emscripten-core#21336)

The TEST_EMSCRIPTEN_SDL_SETEVENTHANDLER macro is copied over from
`test_sdl_key.c` (the SDL1 version) but its not applicable with SDL2.

Remove the 3 second delay from test_sdl_mouse/test_sdl2_mouse.  I ran
these tests many times without any issues.  We can perhaps revive with
something less long but I can't really see how a delay here could be
useful in practice.

* library_idbfs.js: Handle `transaction.onabort` in `reconcile()` (emscripten-core#21326)

If the transaction aborts (which can happen due to, for example, a QuotaExceededError), `onabort` must be handled to ensure that the `callback` actually gets called.

* [wasm64] Run all browser tests in 4gb mode (emscripten-core#21334)

* Allow comments in response files. (emscripten-core#21330)

It can be useful to have comments in a response file for exported
symbols to explain what some stuff might be in there for.

* Site: Remove old, unused copy of theme layout (emscripten-core#21338)

This was added in 2014, not used.

* Fix asan.test_main_thread_em_asm_pthread (emscripten-core#21339)

Without this change this tests fails to compile with `em++: error:
-pthread + ALLOW_MEMORY_GROWTH may run non-wasm code slowly`.

This test was recently added in emscripten-core#21292 and this wasn't caught
immediately because we don't run the `asan` suite during CI.

* Site: Fix typos (emscripten-core#21344)

* Add external ports + options support to embuilder (emscripten-core#21345)

* Update SDL2 version from 2.24.2 to 2.26.0 (emscripten-core#21337)

* Move `demangle` helper to library_legacy.js (emscripten-core#21346)

Followup to emscripten-core#21156

We no longer use this function anywhere in emscripten.

* [embind] Minor refactor to use more modern JS features (emscripten-core#21331)

* Unify handling of deprecated settings (emscripten-core#21355)

Unlike `LEGACY_SETTINGS`, deprecated settings can continue to be used
and the values passed in will continue to be honored.

Once folks have stopped using them we can move them to
`LEGACY_SETTINGS`.

* Warn of usage of legacy library symbols (emscripten-core#21357)

* Rebaseline codesize expectations. NFC

* Remove unused src/library_cyberdwarf.js (emscripten-core#21356)

The rest of the cyberdwarf support was removed back in emscripten-core#12195

* Mark 3.1.54 as released (emscripten-core#21358)

* [test] Simplify new KeyboardEvent calls in test_browser.py. NFC (emscripten-core#21351)

Closure knows about these property bags so there is no need to quote
the keys.

* Consolidate port loading code + embuilder documentation (emscripten-core#21353)

* Always free allocaed memory in mmunmap. (emscripten-core#21363)

Without this change we were returning early from `__syscall_munmap`
if `_munmap_js` failed.  This could happen if `fd` was no longer a valid
open file descriptor.  However, even in this case we still want to go
ahead and free any allocated memory.

Fixes: emscripten-core#21360

* Use `isfile` rather than `exists` when checking for config files (emscripten-core#21361)

This avoids trying parse non-files.

Also, explicitly log when we are not using a config file at all.

* Rebaseline codesize expectations. NFC

* Avoid malloc dependency in tzset (emscripten-core#21375)

* Update glfw port to 1.0.5 (emscripten-core#21369)

This update includes a fix for a memory corruption bug.

* [test] Add EMTEST_SKIP_NETWORK_TESTS (emscripten-core#21380)

I was doing some work on an airplane over the weekend and I found this
useful.

* Feedback from emscripten-core#21380

I accidentally hit submit before applying this change.

* Clarify MIN_CHROME_VERSION also applies to modern Edge [NFC] (emscripten-core#18222)

* Fix python error when building with audio worklets enabled (emscripten-core#21340)

MIN_EDGE_VERSION was removed (emscripten-core#20881), so we do not need it in the
feature matrix.

* Cleanup cpuprofiler.js. NFC (emscripten-core#21387)

- Use mathod syntax
- Cleanup long line

* Fix typo in embind.js (emscripten-core#21384)

nonexistant ==> nonexistent

Fixes: emscripten-core#21372

* Avoid calling addCxaCatch when not linking as C++ (emscripten-core#21386)

For the test I was unable to figure out a way to generate a call to
`__cxa_find_matching_catch` withing hand coding it like this.

This doesn't fix emscripten-core#21381 but it makes the error message way less
confusing.

* Add changelog entry for SDL2 update. NFC (emscripten-core#21390)

Followup to emscripten-core#21337

* Update sdl2-mixer from 2.6.0 to 2.8.0 (emscripten-core#21391)

* [test] Remove redundante/broken call to glTexImage2D. NFC (emscripten-core#21394)

This line was fixed/patched in 2736a26 but I think the old line
was mistakenly left in.

* Simplify test_gl_subdata and test_float_tex. NFC (emscripten-core#21393)

Also, remove test_std_cout_new.cpp completely.  This test seems to
be based on the code from these other two tests but doesn't seem to
server any useful purpose anymore.  Its was first added in ad285f6.

* [Docs] Fix typos [NFC] (emscripten-core#21395)

* Rebaseline codesize expectations. NFC

* Run test_subdata in webgl2 mode. (emscripten-core#21398)

This involved improving the emulation in glTexSubImage2D.

Sadly I needed to disable this new tests in 4gb+ mode since it runs
into a chromium bug.

* file_packager.py: Handle failure to get cached package (emscripten-core#21328)

In certain scenarios (such as when quota is exceeded), it's possible for a metadata entry to exist, but the package data to be missing - and `event.target.result` will be undefined.

Check for and properly handle this by calling `errback`, so that the `preloadFallback` gets triggered.

* [test] Simplify test_openjpeg. NFC (emscripten-core#21401)

This change simplifies the test in 2 ways:

1. Remove the "extra testing" part of the test that tries to run the
   test again if `ALLOW_MEMORY_GROWTH` is set.  The only test modes that
   have `ALLOW_MEMORY_GROWTH` are the santizier modes and it seems odd
   for an individual test to decide to disable this.
2. Remove the alternate `is_sanitizing` method of running the test.
   I believe this was only needed becuase of (1), and so can be safely
   removed.

By removing this complexity this test now has less nesting and is more
directly readable and more similar to other tests.

* Parameterize test_webgpu_compiletest. NFC (emscripten-core#21402)

* Fix webgl tracing issues (emscripten-core#21389)

The webgl tracing mechanism was assuming a fixed number of arguments for
a given function whereas some webgl function can take variable number of
arguments.

Fix this by using the rest operator.  This is not as slow as using
`arguments`.  Also, performance is not really an issue here since we are
about to serialize all the arguments to string and send to the console
which will vastly out weight the cost of using spread here.

I believe the comments here about hot and cold functions as well as the
comment at about the cost of using `arguments` were copied from the
cpuprofiler.js but they don't apply here.

Also, avoid serializing the entire heap, which can cause chrome to hang.

* [test] Add `reftest` helper method for browser tests. NFC (emscripten-core#21392)

This makes it easy distinguish reference tests from other btests.

* Fix EM_JS macro in the face of recent llvm change (emscripten-core#21403)

A recent change to llvm (llvm/llvm-project#81539)
means that the `used` attribute on data forces the segment to be
present in the output file, even if it's not used.

The usage here in `em_js.h` was assuming that the data was not incldued
in the final output.  This change makes the symbol non-static instead
of used, which is enough to force clang to keep the data around but
not the linker (which is exactly the effect we want).

* Fix dup in nodefs by refcounting nodefs file descriptors (emscripten-core#21399)

* Mark test_itimer_pthread as flaky. NFC (emscripten-core#21404)

This should have been marked as such when the other itimer tests were.

* Rebaseline codesize expectations. NFC

* Use JS class for FSStream, FSNode and LazyUint8Array. NFC (emscripten-core#21406)

Converting each of these to classes led to code size savings.

* Added a way to get more information a port (emscripten-core#21376)

* Fix test_openjpeg under wasm64. NFC (emscripten-core#21408)

openjpeg was assuming a different signature for getopt_long which
was not problem under wasm32.

* [test] Move `requires_wasm2js` to `test/common.js`. NFC (emscripten-core#21412)

* Fix some closure compile warnings in JS library code (emscripten-core#21385)

* Fix for resize events when building with ASSERTIONS (emscripten-core#21362)

Building with ASSERTIONS is failing the safe heap checks when e.detail is undefined.

* [emval] Reduce C++ -> JS calls in emscripten::val lifetime management (emscripten-core#21366)

Related to emscripten-core#21300, but some obviously good incremental wins to reduce the cost of ref counting:

1. Have C++ consistently avoid inc/dec for the special reserved values that don't need to be counted, this saves decref calls every time such a `val` object goes out of scope.
2. Add an rvalue reference version of toWireType for `emscripten::val` that can transfer ownership to JS. This saves one call to incref and one call to decref for the case of a c++ function with return type `emscripten::val`  

The cost seems to be single-digit bytes to the WASM.

* [test] Make use of `is_wasm2js` helper (emscripten-core#21413)

Also, don't check for `WASM2JS` settings.  That is a private setting
that cannot be set on the command line.

* Feedback from emscripten-core#21406. NFC (emscripten-core#21415)

* Fix interactive SDL1 tests (emscripten-core#21420)

* [ports] upgrade SDL to 2.28.4 (emscripten-core#20417)

* embind - Remove node exception handling flag for node 17+. (emscripten-core#21411)

Fixes: emscripten-core#21405

* Remove DEMANGLE_SUPPORT from docs (NFC) (emscripten-core#21430)

* Revert "Proxy webgpu calls back to the main thread. NFC (emscripten-core#20124)" (emscripten-core#21388)

This is a partial revert of 992d1e9

* [emrun] Use GNU command line flag names (emscripten-core#21424)

For example `--no-browser` vs `--no_browser`.

Continue to support the old style but don't advertise these.

* Remove IE workaround in jsStackTrace (emscripten-core#21431)

* Extract getIncludeFile function. NFC (emscripten-core#21425)

* Round down any canvas-relative mouse event coordinates (fixes SAFE_HEAP asserts) (emscripten-core#21429)

When using a custom target the coordinates may be doubles, which causes SAFE_HEAP to assert. These are truncated to avoid this (which is what happens anyway, only now it's explicit).

* Cleanup src/library_sdl.js. NFC (emscripten-core#21416)

- Wrap long lines
- Format comments
- Use method syntax
- Remove unnecessary `SDL_audio`/`SDL_unicode`/`SDL_ttfContext` accessor
  functions.

* Remove remaining uses of `eval` in the JS compiler (emscripten-core#21423)

Use `vm.runInCurrentContext` instead.

We already use this in the preprocessor, but not we also use it for
macro expansion and loading of the compiler code itself.

This change should facilitate future changes to the JS compiler, such
as changing the compiler itself to use JS modules.

* Run test_metadce_minimal in STRICT mode. NFC (emscripten-core#21438)

* Rebaseline codesize expectations. NFC

* [Docs] Document how to build and view the website locally (emscripten-core#21434)

* Allow HEAPX symbols in EXPORT_RUNTIME_METHODS (emscripten-core#21439)

- Factor out updateMemoryViews.
- Enable HEAPXX exports to be specified in EXPORT_RUNTIME_METHODS.
- Add all HEAP symbosl to EXPORT_RUNTIME_METHODS unless in STRICT mode.

Replaces emscripten-core#21407

* Add name section and object symbol table support to emsymbolizer (emscripten-core#21367)

With llvm/llvm-project#82083, llvm-symbolizer works
correctly with name sections, so emsymbolizer can use it. Also do the
same for object files with symbol tables.
Object files do not currently work correctly in llvm-symbolizer where code addresses overlap with
data addresses; when that is fixed, emsymbolizer will start working, but we should add a test for it.

* Add some comments regarding __c_longjmp. NFC (emscripten-core#21442)

* [wasm2js] Remove all handling for external memory file (emscripten-core#21217)

We have a lot of support code and complexity in emscripten for handling
of external memory files.  However, its only ever used in wasm2js mode
which is a legacy mode.  The only reason we continue to support it here
IIUC is because its slightly more space efficient than embedding the
data as base64.

For small programs like hello_world this is an over codesize win.  For
larger programs there is a regression in overall size in proportion to
the amount of static data in the program.

* Limit `isCurrentContextWebGL2` usage. NFC (emscripten-core#21444)

I observed that this helper macro is only needed when webgl2 is
supported.

In fact, it would be a bug if it was used when webgl2 was not supported.

* Update expected binaryen version to 116 (emscripten-core#21446)

After WebAssembly/binaryen#6358, we need to
update this to pass the tests.

* Update val.h.rst (emscripten-core#21449)

Update typo in EM_ASYNC_JS code sample

* Update Binaryen version to 117 (emscripten-core#21451)

After WebAssembly/binaryen#6358, I tried to
update this to 117, but it looks due to some unfinished builds I ended
up updating it to 116
(emscripten-core#21446 (comment)).
This brings it up to 117.

* [circleci] Create `run-tests-firefox` task. NFC (emscripten-core#21452)

This allows use to create different firefox test runs, just like
we already can for chrome.

* Update test_sdl_canvas (emscripten-core#21453)

When I run this test locally in firefox I was getting 2785 as the
sum value.  I believe this is because the test includes SDL1 font
rendering which can depend on the system fonts you have installed.

* Minor library_webgl.js cleanup (emscripten-core#21457)

Split out from emscripten-core#21445

* Another library_webgl.js cleanup. NFC (emscripten-core#21458)

Similar to emscripten-core#21457 but this one actually as a code size saving

* Avoid garbage-free WebGL APIs when memory size is over 2gb. (emscripten-core#21445)

Both chrome and firefox see have some issues with passing 2gb+ and 4gb+
offsets to these APIs.

Once the browser issues are addressed we can lift these restrictions
over time.

Fixes: emscripten-core#20533

* Update GLFW contrib port to v1.1.0 (GLFW 3.4.0) (emscripten-core#21455)

On Feb 23rd 2024, GLFW released a new version 3.4.0. This is the changes in contrib.glfw3 (https://github.com/pongasoft/emscripten-glfw) to implement the new apis/features added to 3.4

Release notes for the port: https://github.com/pongasoft/emscripten-glfw/releases/tag/v1.1.0

* Initial new `INITIAL_HEAP` setting (emscripten-core#21071)

Changes in default behavior:
1) INITIAL_HEAP is the new default for most builds.
   This means that there is an increase in the effective
   initial memory used by "sizeof(stack) + sizeof(static data)".
   In typical small applications this should be on the order
   of half a megabyte.
2) Because we cannot precisely calculate the amount
   of initial memory now, ASAN support will use
   the conservative upper estimate of MAXIMUM_MEMORY.
   This only affects ALLOW_MEMORY_GROWTH=0 builds.

This change does not yet enable INITIAL_HEAP for builds
that instantiate the memory in JS, e. g. with threading.

* Fix file:// protocol not working in Ports.fetch_project (emscripten-core#21456)

* Mark 3.1.55 as released (emscripten-core#21461)

* Improve ES6 support in TypeScript definitions generation (emscripten-core#21460)

Users building with --emit-tsd and the EXPORT_ES6 setting should get
TypeScript definitions accounting for the fact that ES6 modules
include a default export that is a functor to instantiate and access
the wasm module in question

When generating TS definitions, query the EXPORT_ES6 setting and,
if present, add the appropriate exports to the output

Include options param in default functor export for `--emit-tsd`.
This serves as a starting point for incluing all options such as
`locateFile` (and the slew of others), without requiring manually
iterating through them

Update tests to include factory function

* Fix Getting WebGPU Context in pthread (emscripten-core#21467)

Failed to run compiled WGPU code clips:
 wgpu::SurfaceDescriptor surfDesc{};
 surfDesc.nextInChain = &canvasDesc;
 wgpu::Surface surface = instance.CreateSurface(&surfDesc);

The reason is that it cannot call getContext from correct canvas
object.

This PR fixed the problem and add correct build flags for
webgpu_basic_rendering test in pthread.

* Honor WEBGL_USE_GARBAGE_FREE_APIS (emscripten-core#21462)

Followup to emscripten-core#21445.  I overlooked some of the webgl2-only paths.

* Rebaseline codesize expectations. NFC

* [test] Disable asan.test_externref_emjs_dynlink. NFC (emscripten-core#21490)

This test is failing due to a crash in LLVM that was introduced
in llvm/llvm-project#83196.

* [JSPI] - Assert that JSPI is supported by the environment. (emscripten-core#21483)

* Fix test_extended_const after recent binaryen change. NFC (emscripten-core#21492)

See WebAssembly/binaryen#6379

* Fix for longjmp(.., 0) under wasm EH (emscripten-core#21493)

Fixes: emscripten-core#21486

* [wasm64] Enable SDL2 tests under wasm64 (emscripten-core#21489)

I think the recent SDL2 version bump must have fixed the issues we
were previously seeing.  See emscripten-core#21337 and emscripten-core#20417.

* Avoid string property access in library_webgpu.js. NFC (emscripten-core#21454)

* Multiple fixes to EmscriptenOrientationChangeEvent (emscripten-core#21428)

* Add support for "-Ofast" (emscripten-core#21494)

Based on emscripten-core#11884

* Fix broken regex from emscripten-core#21492 (emscripten-core#21498)

* [wasm64] Enable test_sdl_audio_beep_sleep (emscripten-core#21495)

* Warn when setting used on the command line more than once (emscripten-core#21464)

See emscripten-core#19938

* [test] Remove EMTEST_LACKS_THREAD_SUPPORT (emscripten-core#21496)

All the browsers we test against these days support threads

* Convert unsafe_optimizations.js to mjs. NFC (emscripten-core#21504)

* [test] Disable asan.test_externref_emjs. NFC (emscripten-core#21508)

Should have been part of emscripten-core#21490.

* Re-enable test_dlfcn_em_asm. NFC (emscripten-core#21500)

This test was disabled due to EM_ASM not working in side modules but
that was fixed in emscripten-core#18228.

* Remove unused genArgSequence and needsQuoting. NFC (emscripten-core#21505)

The usage of genArgSequence removed in emscripten-core#17420.

The usage of needsQuoting was removed in emscripten-core#17462

* Avoid generating empty statement in receiveI64ParamAsI53 (emscripten-core#21487)

Fixes a few Closure lint warnings. (Though of course there are very many
other Closure warnings.)

* Convert lz4-compress.js to mjs. NFC (emscripten-core#21507)

* Fix acorn-optimizer error when destructuring array with empty elements (emscripten-core#21501)

This fixes cases like this:

function ([, a]) { }

In such arrays it is possible to have empty elements, which must be skipped.

* Run JS linter on src/*.mjs files. NFC (emscripten-core#21510)

As of today all `.mjs` files are part of the compiler itself and don't
contains preproccess macros, so that are safe to be run through linter
tools.

* [test] Add `skipExecIf` which just skips the running of the test (emscripten-core#21499)

This means that even in CI when we might skip certain tests for graphics
or sounds we still get some assurance that the test code compiles.

* Avoid use of port 8088 in socket tests. NFC (emscripten-core#21512)

This port is used by the goma compiler webserver which causes these
tests to fail on machines that are running the goma compiler.

* Remove check for file type of `.a` files. NFC (emscripten-core#21513)

Instead just pass them through the linker like other inputs.

See emscripten-core#21128

* [wasm64] Fix WebSocket API under wasm64 (emscripten-core#21514)

Fixes: emscripten-core#21503

* Only check for WebAssembly support in debug builds. NFC (emscripten-core#21511)

Also, avoid the use of `abort` in this case since abort itself depends
on `WebAssembly.RuntimeError`.

Fixes: emscripten-core#21484

* Fix unconditional usage of `requests` import (emscripten-core#21523)

On systems that don't have the requests python we cannot catch
this exception.

* Use curl to download files on macOS (emscripten-core#21525)

* Use DEPRECATED_SETTINGS for RUNTIME_LINKED_LIBS (emscripten-core#21522)

And also for CLOSURE_WARNINGS

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Brendan Dahl <brendan.dahl@gmail.com>
Co-authored-by: Sam Clegg <sbc@chromium.org>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Filip Lundgren <filip@sceneri.com>
Co-authored-by: Kai Ninomiya <kainino@chromium.org>
Co-authored-by: Alon Zakai <azakai@google.com>
Co-authored-by: Kleis Auke Wolthuizen <github@kleisauke.nl>
Co-authored-by: Xdestiny <bitzyw@126.com>
Co-authored-by: Jerry.Zhuang <jerry.zhuang@jwzg.com>
Co-authored-by: Yan Pujante <yan@pongasoft.com>
Co-authored-by: Bruce Mitchener <bruce.mitchener@gmail.com>
Co-authored-by: MichalGrzymek <mjgrzymek@gmail.com>
Co-authored-by: Simon Cooper <simon.d.cooper@hotmail.co.uk>
Co-authored-by: mrolig5267319 <134731827+mrolig5267319@users.noreply.github.com>
Co-authored-by: kangtastic <942136+kangtastic@users.noreply.github.com>
Co-authored-by: Derek Schuff <dschuff@chromium.org>
Co-authored-by: Piotr Wierciński <laminowany@gmail.com>
Co-authored-by: Dannii Willis <curiousdannii@gmail.com>
Co-authored-by: pastdue <30942300+past-due@users.noreply.github.com>
Co-authored-by: François Hautier <138592079+FrancoisRecisio@users.noreply.github.com>
Co-authored-by: Marcin Wojdyr <wojdyr@gmail.com>
Co-authored-by: Hood Chatham <roberthoodchatham@gmail.com>
Co-authored-by: Carl Woffenden <cwoffenden@gmail.com>
Co-authored-by: Charlie Birks <charlie@daft.games>
Co-authored-by: Érico Porto <ericoporto2008@gmail.com>
Co-authored-by: Heejin Ahn <aheejin@gmail.com>
Co-authored-by: patrick kettner <patrickkettner@gmail.com>
Co-authored-by: SingleAccretion <62474226+SingleAccretion@users.noreply.github.com>
Co-authored-by: J Nicholas Giampietro <jnickg@adobe.com>
Co-authored-by: Yan, Shaobo <shaobo.yan@intel.com>
Co-authored-by: nsfisis <54318333+nsfisis@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sbc100 sbc100 sbc100 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Is it possible avoid binding js to use unsafe-eval syntax?
2 participants
@brendandahl @sbc100