-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix dup*() syscalls #9396
base: main
Are you sure you want to change the base?
Fix dup*() syscalls #9396
Conversation
775edd6
to
6bee509
Compare
Looks like the test is also failing in CI. Really not sure what's going on there. |
If it works locally but fails remotely, one possibility is using a different version of node. I think our CI uses the default emsdk version. I see failures in exceptions tests etc. though, so all of those pass for you locally? |
It works locally when I manually build the test code and run it, but not when using the test runner locally, which is what's weird.
During development, I only ran
I just ran all the wasm0 tests locally on this branch, and the only failing tests are
Whereas it fails when using the runner.
Additionally,
|
If it works when you build it, but not when the test runner runs it, then there is probably a difference in how it's build. You can build with |
Upon taking a closer look at this, it appears the failure is being caused by testing with |
6bee509
to
d1f4ada
Compare
@kripken Well that was fun. It turns out that Node doesn't have the same concept of duplicating file descriptors, so I implemented a shared refcount that keeps the stream open until no more fds are referencing it. The test now passes with both |
fc0e3af
to
355cdff
Compare
If node doesn't support duplicating fds, then that worries me about how useful this will be, I think? How does the proposed refcount facility work - are duplicated fds all refering to the original they were duplicated from? Does that have the right behavior (I think they should not pick up changes to the other fd after the duplication?) |
@kripken It certainly appears to have the right behavior to me, based on the results of the test running natively, in the browser, and in Node. dup(2) describes:
This is implemented in the in-memory filesystem by creating a duplicate stream object containing a shared object that stores the stream's position and status flags. Updating either property on one fd will change it on a duplicated fd, which is demonstrated to be correct by running the provided test. Prior to this patch, duplicated streams did not share seek position and status flags. NODERAWFS is a little different, because opening a file creates a node fd for actual io ops. Closing the node stream will also make duplicated fds unavailable, so upon duplicating an fd, we increment the refcount in the shared object. Closing an fd decrements the refcount, and we only close the stream upon having no more references to it. This also passes the provided test. The one exception described is for the
|
I see @jakogut, sounds good! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work! Some comments, but basically this all looks good.
Also, if the refcount capability is only for NODERAWFS, it would be good to clarify that, and why it is needed there, in a comment - basically, please explain what you explained in the last comment to me, also in the code.
355cdff
to
abb5b77
Compare
By the way, it's been a while since I came across this issue, but if memory serves, FFmpeg uses dup in this way somewhere, and this PR fixes this issue. |
Oops, looks like some of the indenting is wrong, sorry for the noise. (I have to fix my text editor) |
abb5b77
to
89699d9
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! Looks good aside from that indentation error, and we need to get tests passing.
About the tests, odd it fails in sdl2_mixer
- do you see that locally too? Let me know if you need help debugging that.
src/library_fs.js
Outdated
if (sharedProperties.indexOf(p) >= 0) { | ||
if (!("shared" in newStream)) { | ||
newStream.shared = {}; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
indentation looks wrong here
(Btw, we recently fixed some sdl2_mixer issue - merging in latest incoming to here may help, but probably a long shot.) |
I wasn't able to reproduce the embuilder sdl2-mixer issue, but I've fixed the whitespace problem, and rebased on incoming. |
89699d9
to
186fc05
Compare
Look like |
It looks like this test passes when compiled normally, and fails with the closure compiler. EDIT: All optimization options pass, including Here's the output of the failing test with
@kripken Any ideas for how to debug this further? |
186fc05
to
41a5cb3
Compare
I think the test failures here will be fixed by merging latest incoming to here, as that was an llvm update that we fixed meanwhile. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice, thanks! lgtm aside from the test failures as just mentioned.
41a5cb3
to
8adc4e2
Compare
Rebased on incoming. |
Looks like we're still having Freetype failures. |
Very strange... the parent commit you rebased on passed CI, but this doesn't. I don't understand why that is. One thing to try is to do a merge commit into here (the merge commit will get squashed out anyhow) of latest incoming (might need to wait for the next commit there). |
Still no dice. I checked out incoming at 78071f9, and reproduced the failure locally running $ >> python tests/runner.py wasm0.test_freetype
tests/runner.py:WARNING: use EMTEST_ALL_ENGINES=1 in the env to run against all JS engines, which is slower but provides more coverage
Test suites:
['test_core']
Running test_core: (1 tests)
(checking sanity from test runner)
shared:INFO: (Emscripten: Running sanity checks)
test_freetype (test_core.wasm0) ... <building and saving freetype_O0_069aaf9e9c5b51519ef5cf1a62791478 into cache>
-- configure stdout --
FreeType build system -- automatic system detection
The following settings are used:
platform unix
compiler /home/joseph/emscripten/emcc
configuration directory ./builds/unix
configuration rules ./builds/unix/unix.mk
If this does not correspond to your system or settings please remove the file
`config.mk' from this directory then read the INSTALL file for help.
Otherwise, simply type `make' again to build the library,
or `make refdoc' to build the API reference (the latter needs python).
Generating modules list in ./objs/ftmodule.h...
* module: truetype (Windows/Mac font files with extension *.ttf or *.ttc)
* module: type1 (Postscript font files with extension *.pfa or *.pfb)
* module: cff (OpenType fonts with extension *.otf)
* module: cid (Postscript CID-keyed fonts, no known extension)
* module: pfr (PFR/TrueDoc font files with extension *.pfr)
* module: type42 (Type 42 font files with no known extension)
* module: winfnt (Windows bitmap fonts with extension *.fnt or *.fon)
* module: pcf (pcf bitmap fonts)
* module: bdf (bdf bitmap fonts)
* module: sfnt (helper module for TrueType & OpenType formats)
* module: autofit (automatic hinting module)
* module: pshinter (Postscript hinter module)
* module: raster (monochrome bitmap renderer)
* module: smooth (anti-aliased bitmap renderer)
* module: smooth (anti-aliased bitmap renderer for LCDs)
* module: smooth (anti-aliased bitmap renderer for vertical LCDs)
* module: psaux (Postscript Type 1 & Type 2 helper module)
* module: psnames (Postscript & Unicode Glyph name handling)
done.
cd builds/unix; ./configure '--disable-shared' '--without-zlib'
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking for gcc... /home/joseph/emscripten/emcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... TypeError: WebAssembly Instantiation: Import #3 module="wasi_unstable" error: module is not an object or function
RuntimeError: abort(TypeError: WebAssembly Instantiation: Import #3 module="wasi_unstable" error: module is not an object or function) at Error
at jsStackTrace (/tmp/emscripten_test_wasm0_kmjz40re/building/freetype/builds/unix/conftest:1800:17)
at stackTrace (/tmp/emscripten_test_wasm0_kmjz40re/building/freetype/builds/unix/conftest:1817:16)
at abort (/tmp/emscripten_test_wasm0_kmjz40re/building/freetype/builds/unix/conftest:1601:44)
at /tmp/emscripten_test_wasm0_kmjz40re/building/freetype/builds/unix/conftest:1720:7
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:189:7)
at Function.Module.runMain (module.js:696:11)
at startup (bootstrap_node.js:204:16)
at bootstrap_node.js:625:3
-- end configure stdout --
-- configure stderr --
configure: error: in `/tmp/emscripten_test_wasm0_kmjz40re/building/freetype/builds/unix':
configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details
make: *** [builds/unix/detect.mk:84: setup] Error 1
-- end configure stderr --
ERROR
======================================================================
ERROR: test_freetype (test_core.wasm0)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/joseph/emscripten/tests/runner.py", line 139, in decorated
return func(self, *args, **kwargs)
File "/home/joseph/emscripten/tests/test_core.py", line 5946, in test_freetype
libraries=self.get_freetype_library(),
File "/home/joseph/emscripten/tests/runner.py", line 1187, in get_freetype_library
return self.get_library('freetype', os.path.join('objs', '.libs', 'libfreetype.a'), configure_args=['--disable-shared', '--without-zlib'])
File "/home/joseph/emscripten/tests/runner.py", line 909, in get_library
return build_library(name, build_dir, output_dir, generated_libs, configure,
File "/home/joseph/emscripten/tests/runner.py", line 1690, in build_library
Building.configure(configure + configure_args, env=env, stdout=stdout, stderr=stderr)
File "/home/joseph/emscripten/tools/shared.py", line 1722, in configure
run_process(args, stdout=stdout, stderr=stderr, env=env)
File "/home/joseph/emscripten/tools/shared.py", line 184, in run_process
ret = subprocess.run(cmd, check=check, input=input, *args, **kw)
File "/usr/lib/python3.8/subprocess.py", line 512, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['sh', './configure', '--disable-shared', '--without-zlib']' returned non-zero exit status 2.
----------------------------------------------------------------------
Ran 1 test in 6.196s
FAILED (errors=1) |
Interesting - @tlively saw that error locally a few days ago, but we don't remember what made it go away... Perhaps try clearing the cache, or install the latest tot build? Also, do other tests pass for you locally (like |
Rework dup syscalls so file descriptors share all flags and also seek position.
4dfca0b
to
b278cbb
Compare
Other tests pass, including Clearing the cache doesn't change anything. I'm using the latest version of the |
Ah, if I remember correctly I saw that error when I had just pulled a new version of Binaryen but forgot to build it. Anyway, there is no guarantee that the latest versions of LLVM, Binaryen, and Emscripten work together correctly (and it is expected that sometimes they do not). I highly recommend using emsdk to install known-good combinations of tools or if you must manage the tools yourself, look at https://chromium.googlesource.com/emscripten-releases/ to see what emsdk's tot tag is shipping. |
The deletion of the Re-opening and re-targeting to master. |
This issue has been automatically marked as stale because there has been no activity in the past year. It will be closed automatically if no further activity occurs in the next 30 days. Feel free to re-open at any time if this issue is still relevant. |
Any chance of rebasing/landing this? I just ran into this issue as well. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This LGTM modulo the comment about ERRNO_CODES -> cDefine
@@ -1310,8 +1315,13 @@ var SyscallsLibrary = { | |||
#if ASSERTIONS | |||
assert(!flags); | |||
#endif | |||
if (old.fd === suggestFD) return -{{{ cDefine('EINVAL') }}}; | |||
return SYSCALLS.doDup(old.path, old.flags, suggestFD); | |||
if (old.fd === suggestFD) return -ERRNO_CODES.EINVAL; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
switch these to use the cDefine to avoid a dep on ERRNO_CODES which, I believe was removed
I can see about rebasing this later and applying the above feedback. |
@jakogut do you have time to look at rebasing this change? |
Are you going to resolve this conflict soon to merge changes? I see that changes are approved. |
I think a large part of this change landed in #14808.. but a rebase to see if there is anything else worth landing seems like a good idea. |
Rework dup syscalls so file descriptors share all flags and also seek position.
Based on #4017
Rebased and modified for sharing flags and position, but not file descriptors or closed status.
Closes #4017
NOTE: The included test passes for me when building and running manually, but fails using the runner.