Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

src: use simdutf for converting externalized builtins to UTF-16 #46119

Merged

Conversation

addaleax
Copy link
Member

@addaleax addaleax commented Jan 6, 2023

Remove the dependency on ICU for this part, as well as the hacky way of converting embedder main sources to UTF-8 via V8 APIs. Allow UnionBytes to own the memory its pointing to in order to simplify the code on the BuiltinLoader side.

Remove the dependency on ICU for this part, as well as the
hacky way of converting embedder main sources to UTF-8 via
V8 APIs. Allow `UnionBytes` to own the memory its pointing
to in order to simplify the code on the `BuiltinLoader` side.
@nodejs-github-bot
Copy link
Collaborator

Review requested:

  • @nodejs/gyp
  • @nodejs/startup

@nodejs-github-bot nodejs-github-bot added lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels Jan 6, 2023
@addaleax addaleax added the request-ci Add this label to start a Jenkins CI on a PR. label Jan 6, 2023
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Jan 6, 2023
@nodejs-github-bot

This comment was marked as outdated.

Copy link
Member

@anonrig anonrig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perfect. lgtm.

@addaleax addaleax added request-ci Add this label to start a Jenkins CI on a PR. commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. labels Jan 6, 2023
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Jan 6, 2023
@nodejs-github-bot

This comment was marked as outdated.

@addaleax addaleax added the request-ci Add this label to start a Jenkins CI on a PR. label Jan 6, 2023
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Jan 6, 2023
@nodejs-github-bot
Copy link
Collaborator

@anonrig anonrig added author ready PRs that have at least one approval, no pending requests for changes, and a CI started. review wanted PRs that need reviews. labels Jan 7, 2023
@addaleax addaleax added the c++ Issues and PRs that require attention from people who are familiar with C++. label Jan 9, 2023
@addaleax
Copy link
Member Author

addaleax commented Jan 9, 2023

@nodejs/cpp-reviewers Anybody else want to take a look here?

utf8source.data(),
utf8source.length(),
reinterpret_cast<char16_t*>(out->data()));
out->resize(u16_length);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this bit really necessary?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What kind of alternative would you suggest? Or is this about the resize() call?

Add(id, source);
}

bool BuiltinLoader::Add(const char* id, std::string_view utf8source) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we can just create a UnionBytes constructor that takes a std::string_view that does this? (then I guess that UnionBytes can own the vector?))

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, UnionBytes that supports storing UTF-8 directly?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fwiw, I’ll merge this in order to unblock #45942, but I’m still happy to iterate further here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A union for everything ;)

@addaleax addaleax added the commit-queue Add this label to land a pull request using GitHub Actions. label Jan 10, 2023
@nodejs-github-bot nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Jan 10, 2023
@nodejs-github-bot nodejs-github-bot merged commit 21fb98e into nodejs:main Jan 10, 2023
@nodejs-github-bot
Copy link
Collaborator

Landed in 21fb98e

@addaleax addaleax deleted the simdutf-for-builtin-conversions branch January 10, 2023 11:26
RafaelGSS pushed a commit to RafaelGSS/node that referenced this pull request Jan 17, 2023
Remove the dependency on ICU for this part, as well as the
hacky way of converting embedder main sources to UTF-8 via
V8 APIs. Allow `UnionBytes` to own the memory its pointing
to in order to simplify the code on the `BuiltinLoader` side.

PR-URL: nodejs#46119
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
RafaelGSS pushed a commit that referenced this pull request Jan 20, 2023
Remove the dependency on ICU for this part, as well as the
hacky way of converting embedder main sources to UTF-8 via
V8 APIs. Allow `UnionBytes` to own the memory its pointing
to in order to simplify the code on the `BuiltinLoader` side.

PR-URL: #46119
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
@RafaelGSS RafaelGSS mentioned this pull request Jan 20, 2023
juanarbol pushed a commit that referenced this pull request Jan 26, 2023
Remove the dependency on ICU for this part, as well as the
hacky way of converting embedder main sources to UTF-8 via
V8 APIs. Allow `UnionBytes` to own the memory its pointing
to in order to simplify the code on the `BuiltinLoader` side.

PR-URL: #46119
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
@juanarbol juanarbol mentioned this pull request Jan 28, 2023
juanarbol pushed a commit that referenced this pull request Jan 31, 2023
Remove the dependency on ICU for this part, as well as the
hacky way of converting embedder main sources to UTF-8 via
V8 APIs. Allow `UnionBytes` to own the memory its pointing
to in order to simplify the code on the `BuiltinLoader` side.

PR-URL: #46119
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
codebytere added a commit to electron/electron that referenced this pull request Feb 6, 2023
codebytere added a commit to electron/electron that referenced this pull request Feb 6, 2023
codebytere added a commit to electron/electron that referenced this pull request Feb 6, 2023
jkleinsc pushed a commit to electron/electron that referenced this pull request Feb 7, 2023
* chore: bump node in DEPS to v18.14.0

* src: add support for externally shared js builtins

nodejs/node#44376

* test: fix test broken under --node-builtin-modules-path

nodejs/node#45894

* build: add option to disable shared readonly heap

nodejs/node#45887

* src: remove unnecessary semicolons

nodejs/node#46171

* src: remove return after abort

nodejs/node#46172

* chore: fixup patch indices

* test_runner: parse yaml

nodejs/node#45815

* src: use simdutf for converting externalized builtins to UTF-16

nodejs/node#46119

* src: rename internal module declaration as internal bindings

nodejs/node#45551

---------

Co-authored-by: electron-roller[bot] <84116207+electron-roller[bot]@users.noreply.github.com>
Co-authored-by: Shelley Vohr <shelley.vohr@gmail.com>
khalwa pushed a commit to solarwindscloud/electron that referenced this pull request Feb 22, 2023
* chore: bump node in DEPS to v18.14.0

* src: add support for externally shared js builtins

nodejs/node#44376

* test: fix test broken under --node-builtin-modules-path

nodejs/node#45894

* build: add option to disable shared readonly heap

nodejs/node#45887

* src: remove unnecessary semicolons

nodejs/node#46171

* src: remove return after abort

nodejs/node#46172

* chore: fixup patch indices

* test_runner: parse yaml

nodejs/node#45815

* src: use simdutf for converting externalized builtins to UTF-16

nodejs/node#46119

* src: rename internal module declaration as internal bindings

nodejs/node#45551

---------

Co-authored-by: electron-roller[bot] <84116207+electron-roller[bot]@users.noreply.github.com>
Co-authored-by: Shelley Vohr <shelley.vohr@gmail.com>
gecko19 pushed a commit to brightsign/electron that referenced this pull request Feb 28, 2023
* chore: bump node in DEPS to v18.14.0

* src: add support for externally shared js builtins

nodejs/node#44376

* test: fix test broken under --node-builtin-modules-path

nodejs/node#45894

* build: add option to disable shared readonly heap

nodejs/node#45887

* src: remove unnecessary semicolons

nodejs/node#46171

* src: remove return after abort

nodejs/node#46172

* chore: fixup patch indices

* test_runner: parse yaml

nodejs/node#45815

* src: use simdutf for converting externalized builtins to UTF-16

nodejs/node#46119

* src: rename internal module declaration as internal bindings

nodejs/node#45551

---------

Co-authored-by: electron-roller[bot] <84116207+electron-roller[bot]@users.noreply.github.com>
Co-authored-by: Shelley Vohr <shelley.vohr@gmail.com>
gecko19 pushed a commit to brightsign/electron that referenced this pull request Mar 15, 2023
* chore: bump node in DEPS to v18.14.0

* src: add support for externally shared js builtins

nodejs/node#44376

* test: fix test broken under --node-builtin-modules-path

nodejs/node#45894

* build: add option to disable shared readonly heap

nodejs/node#45887

* src: remove unnecessary semicolons

nodejs/node#46171

* src: remove return after abort

nodejs/node#46172

* chore: fixup patch indices

* test_runner: parse yaml

nodejs/node#45815

* src: use simdutf for converting externalized builtins to UTF-16

nodejs/node#46119

* src: rename internal module declaration as internal bindings

nodejs/node#45551

---------

Co-authored-by: electron-roller[bot] <84116207+electron-roller[bot]@users.noreply.github.com>
Co-authored-by: Shelley Vohr <shelley.vohr@gmail.com>
gecko19 pushed a commit to brightsign/electron that referenced this pull request Mar 15, 2023
* chore: bump node in DEPS to v18.14.0

* src: add support for externally shared js builtins

nodejs/node#44376

* test: fix test broken under --node-builtin-modules-path

nodejs/node#45894

* build: add option to disable shared readonly heap

nodejs/node#45887

* src: remove unnecessary semicolons

nodejs/node#46171

* src: remove return after abort

nodejs/node#46172

* chore: fixup patch indices

* test_runner: parse yaml

nodejs/node#45815

* src: use simdutf for converting externalized builtins to UTF-16

nodejs/node#46119

* src: rename internal module declaration as internal bindings

nodejs/node#45551

---------

Co-authored-by: electron-roller[bot] <84116207+electron-roller[bot]@users.noreply.github.com>
Co-authored-by: Shelley Vohr <shelley.vohr@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
author ready PRs that have at least one approval, no pending requests for changes, and a CI started. c++ Issues and PRs that require attention from people who are familiar with C++. commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. review wanted PRs that need reviews.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants