Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(node): seperate worker module cache #23634

Merged
merged 11 commits into from May 16, 2024

Conversation

littledivy
Copy link
Member

Construct a new module graph container for workers instead of sharing it with the main worker.

Fixes #17248
Fixes #23461

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How's the memory usage? Does it reuse the Arc<str> when it can? (ex. for remote modules)

Also, what happens with --reload?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@littledivy do you think we could reduce memory usage by sharing the strings in memory when they're the same?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, maybe we could store specifier -> hash mapping in a shared container and compare that in the file fetcher. Doing this check without comparing specifiers first may be expensive but more effective in some cases.

@@ -0,0 +1,4 @@
{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: it would be nice to convert this test into a step under worker_threads folder instead of creating worker_threads_cache folder.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm. I thought it would be clean to convert several tests of a module into steps but it will break reporting.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After #23667, you can now declare serveral tests in __test__.jsonc itself.

cli/module_loader.rs Outdated Show resolved Hide resolved
Copy link
Member

@bartlomieju bartlomieju left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Drive-by: can we add tests for both Web and Node workers that verify this behavior?

@dsherret
Copy link
Member

dsherret commented May 4, 2024

Looks like some wpt tests are failing now.

@littledivy
Copy link
Member Author

littledivy commented May 4, 2024

Looks like some wpt tests are failing now.

It's a new test thats passing now.

 "/html/semantics/scripting-1/the-script-element/module/dynamic-import/blob-url-workers.window.html - A revoked blob URL will not resolve in a worker even if it's in the window's module graph"

@bartlomieju
Copy link
Member

Copy link
Member

@dsherret dsherret left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

@dsherret dsherret left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Seems like there's some errors after merge though.

(We will circle back to add a string cache later)

@littledivy littledivy enabled auto-merge (squash) May 16, 2024 06:26
@littledivy littledivy merged commit 88983fb into denoland:main May 16, 2024
17 checks passed
bartlomieju pushed a commit that referenced this pull request May 16, 2024
Construct a new module graph container for workers instead of sharing it
with the main worker.

Fixes #17248
Fixes #23461

---------

Co-authored-by: David Sherret <dsherret@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

node:worker_threads module cache is shared Support for SvelteKit
4 participants