-
Notifications
You must be signed in to change notification settings - Fork 29k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
src: add public API to create isolate and context #20639
Conversation
Ping @nodejs/n-api ... would it make sense to have n-api support for this? |
i'm not quite sure how we would be able to represent contexts in napi but something like napi_create_env would probably work for isolates |
@jasnell I think it’s going to make sense to work out and eventually provide an N-API-style embedder API for Node.js, yes. But this makes sense regardless, and we’d probably put more thoughts into a whole new API that we can design from scratch. |
I was thinking something as simple as napi_env napi_create_env(); ... and then you can use Now, if only we could figure out how not to leak the dlopen handles. |
... or napi_status napi_create_env(napi_env *result); in keeping with our existing API look and feel. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Besides the bug, I also have some misgivings about haphazardly exposing yet another piece of internal plumbing.
It doesn't result in a coherent public API but meanwhile makes maintenance harder.
src/node.cc
Outdated
@@ -279,6 +279,8 @@ static v8::Isolate* node_isolate; | |||
|
|||
DebugOptions debug_options; | |||
|
|||
static ArrayBufferAllocator array_buffer_allocator; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This can't be static
, it's tied to the isolate. That will be problematic because of the zeroFill
field - that field is flipped on and off at run-time and that's observable (read: behaves erratically) between isolates.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@bnoordhuis zeroFill is tied with this condition if (auto zero_fill_field = env->isolate_data()->zero_fill_field()). Currently, the API of CreateIsolateData doesn't support parameter of zero_fill_field for embedding/addon users. It means zeroFill is only tied to the node main thread/environment. This is my understanding at first, and I thought it was not a bad design of node public API.
I can change it to let each isolate have its own ArrayBufferAllocator instance if you think the above understanding of the current node APIs doesn't work or just not right.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@helloshuangzi In that case, it might make sense to expose a way to create a Node.js-style ArrayBufferAllocator
given an IsolateData*
pointer?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@helloshuangzi here is why it's not safe:
Lines 115 to 122 in 46d335c
function createUnsafeArrayBuffer(size) { | |
zeroFill[0] = 0; | |
try { | |
return new ArrayBuffer(size); | |
} finally { | |
zeroFill[0] = 1; | |
} | |
} |
That's the 'flipped on and off at run-time' I mentioned that is observable from another isolate.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@bnoordhuis , Yes, but 'zeroFill' is only available in node main isolate/environment because of if (auto zero_fill_field = env->isolate_data()->zero_fill_field()) based on current node API.
I am thinking that it might be not a bad idea to make all isolates comply with node main isolate/environment about zero_fill_field。
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, that's what I said all isolates will comply with node main isolate about setting of zero_fill_field, while only node main isolate can change it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, that's what I said all isolates will comply with node main isolate about setting of zero_fill_field, while only node main isolate can change it.
I don’t think that’s a good idea. I agree with Ben, it’s not a good idea for one Isolate to be able to change the allocation mode of other Isolates.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To reinforce the point: what you'd see in other isolates is new arraybuffers randomly being filled with garbage instead of zeroes like they're supposed to.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay, @addaleax and @bnoordhuis, thanks for the feedback about this point. Let me change it to give an ArrayBufferAllocator instance for each isolate.
@addaleax, as you commented above, we will need:
- an API to create node.js-style ArrayBufferAllocator, and get its zero_fill_filed.
- a new API of CreateIsolateData which can take the parameter of zero_fill_field.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed, please help review!
@devsnek the |
I share @bnoordhuis's concern for creating new public API without the complete picture. I also understand it may not be helpful to get in the way when it may take a while to come up with a complete API. @helloshuangzi I'm wondering how urgent this is from your perspective? Is it significantly blocking your forward progress or something that is a nice to have in the short term? |
@mhdawson and @bnoordhuis , thanks for your feedback. @mhdawson, thanks for your asking about the urgency. We are working on re-implementing napajs, a node addon for js mutli-threading based on node public APIs (before we do most work by ourselves). Unfortunately, it is a blocker for this work. |
@helloshuangzi Can you explain in what way it is a blocker? Generally, one should be able to get by using only the V8 APIs for creating Isolates + Contexts, so that seems like helpful information. (Like you said, the APIs introduced here would be “helpers”, and I am not sure in what way they are necessary themselves, tbh.) |
@addaleax isolates that aren't tracked by node's platform tend to screwed over pretty badly through the lifetime. i think there's another pr to expose node's platform because of that. |
@devsnek That’s not really related to this PR – |
@addaleax, I say it's a blocker because we are making every effort to maximize the consistency between napa worker and node default isolate/environment (or say it a node standard way to create isolate/context). The consistency is important to us to give napa users a smooth experience. That's why we are trying to implement napa based on node public API. Without the proposed APIs, addon/embedding users will always need to catch up with node changes upon this part. |
src/node.cc
Outdated
@@ -4425,15 +4425,22 @@ int EmitExit(Environment* env) { | |||
} | |||
|
|||
|
|||
IsolateData* CreateIsolateData(Isolate* isolate, uv_loop_t* loop) { | |||
return new IsolateData(isolate, loop, nullptr); | |||
ArrayBufferAllocatorBase* CreateArrayBufferAllocator() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we could also use a std::unique_ptr
, and/or drop FreeArrayBufferAllocator
because that would be implicit in using std::unique_ptr
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just followed the pattern of the existing APIs like FreeIsolateData, FreeEnvironment and FreePlatform. I think they were made in the current pattern to avoid trouble of unique_ptr for ABI.
Do you think I can keep this pattern for ArrayBufferAllocator?
src/node.h
Outdated
NODE_EXTERN IsolateData* CreateIsolateData( | ||
v8::Isolate* isolate, | ||
struct uv_loop_s* loop, | ||
MultiIsolatePlatform* platform); | ||
MultiIsolatePlatform* platform = nullptr, | ||
uint32_t* zero_fill_field = nullptr); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just fyi, removal of the first variant and adding the field here would be an ABI breakage and therefore semver-major (adding the label for now). I think you might first want to go the route of adding Yet Another Overload, maybe leaving a TODO()
comment for switching to default parameters in a follow-up PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks for the kind reminder. Just added a new overload to avoid semver-major for this PR, and added TODO().
src/node.h
Outdated
|
||
protected: | ||
uint32_t zero_fill_field_ = 1; // Boolean but exposed as uint32 to JS land. | ||
}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is the intention that we expose zero_fill_field()
for public use? I don’t think so. It might be easier/expose less internal API surface to just forward-declare ArrayBufferAllocator
and add an ArrayBufferAllocator*
parameter to CreateIsolateData()
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You are right, no public use of zero_fill_field. Just updated it as your suggestion.
Oh, and: Please avoid merge commits. :) It would be great if you could rebase them out of this PR; it you’d prefer not to, the person landing this PR would have to do that, which can get a bit messy if there are merge conflicts along the way. |
src/node.cc
Outdated
@@ -4425,10 +4425,21 @@ int EmitExit(Environment* env) { | |||
} | |||
|
|||
|
|||
ArrayBufferAllocator* CreateArrayBufferAllocator() { | |||
return new ArrayBufferAllocator; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
new ArrayBufferAllocator()
- i.e., include parentheses.
src/node.cc
Outdated
inline int Start(uv_loop_t* event_loop, | ||
int argc, const char* const* argv, | ||
int exec_argc, const char* const* exec_argv) { | ||
ArrayBufferAllocator* allocator = CreateArrayBufferAllocator(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use a std::unique_ptr with a custom deleter, or simply use a stack-allocated ArrayBufferAllocator.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just followed the pattern of the existing APIs like FreeIsolateData, FreeEnvironment and FreePlatform. I think they were made in the current pattern to avoid trouble of unique_ptr for ABI.
Do you think I can keep this pattern for ArrayBufferAllocator?
Updated with unique_ptr, and I don't have to customize deleter for usage in node.cc because it can get the full definition here. As to the public API/ABI, do you think I can keep it with raw pointer by following the existing pattern?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, raw pointers are the way to go, that avoids trouble when Node.js and the add-on use different allocators.
src/node.cc
Outdated
@@ -4611,15 +4639,16 @@ inline int Start(uv_loop_t* event_loop, | |||
Locker locker(isolate); | |||
Isolate::Scope isolate_scope(isolate); | |||
HandleScope handle_scope(isolate); | |||
IsolateData isolate_data( | |||
IsolateData* isolate_data = CreateIsolateData( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM with a request.
I want it understood that there's no promise we'll carry these APIs forever. If they become a maintenance hassle, we'll simply remove them again in the next major release.
src/node.cc
Outdated
inline int Start(uv_loop_t* event_loop, | ||
int argc, const char* const* argv, | ||
int exec_argc, const char* const* exec_argv) { | ||
std::unique_ptr<ArrayBufferAllocator> allocator(CreateArrayBufferAllocator()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I prefer simply ArrayBufferAllocator allocator;
if you're not using a custom deleter, no point in heap-allocating it. Likewise for IsolateData
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the detail description. I just want to reuse the APIs to help keep consistency between node-main-thread and addons when the relevant logic gets updated in the future.
Updated with custom deleter for both ArrayBufferAllocator and IsolateData.
PR-URL: #20639 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com> Reviewed-By: Khaidi Chu <i@2333.moe> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
PR-URL: nodejs#20639 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com> Reviewed-By: Khaidi Chu <i@2333.moe> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Notable Changes: * **addons**: - Fixed a memory leak for users of `AsyncResource` and N-API. (Michael Dawson) [#20668](#20668) * **assert**: - The `error` parameter of `assert.throws()` can be an object containing regular expressions now. (Ruben Bridgewater) [#20485](#20485) * **crypto**: - The `authTagLength` option has been made more flexible. (Tobias Nießen) [#20235](#20235), [#20039](#20039) * **http**: - Handling of `close` and `aborted` events has been made more consistent. (Robert Nagy) [#20075](#20075), [#20611](#20611) * Embedder support: - Functions for creating V8 `Isolate` and `Context` objects with Node.js-specific behaviour have been added to the API. (helloshuangzi) [#20639](#20639) - Node.js `Environment`s clean up resources before exiting now. (Anna Henningsen) [#19377](#19377) - Support for multi-threaded embedding has been improved. (Anna Henningsen) [#20542](#20542), [#20539](#20539), [#20541](#20541) * **timers**: - `timeout.refresh()` has been added to the public API. (Jeremiah Senkpiel) [#20298](#20298) PR-URL: #20724
Notable Changes: * **addons**: - Fixed a memory leak for users of `AsyncResource` and N-API. (Michael Dawson) [#20668](#20668) * **assert**: - The `error` parameter of `assert.throws()` can be an object containing regular expressions now. (Ruben Bridgewater) [#20485](#20485) * **crypto**: - The `authTagLength` option has been made more flexible. (Tobias Nießen) [#20235](#20235), [#20039](#20039) * **http**: - Handling of `close` and `aborted` events has been made more consistent. (Robert Nagy) [#20075](#20075), [#20611](#20611) * Embedder support: - Functions for creating V8 `Isolate` and `Context` objects with Node.js-specific behaviour have been added to the API. (helloshuangzi) [#20639](#20639) - Node.js `Environment`s clean up resources before exiting now. (Anna Henningsen) [#19377](#19377) - Support for multi-threaded embedding has been improved. (Anna Henningsen) [#20542](#20542), [#20539](#20539), [#20541](#20541) * **esm**: - Builtin modules (e.g. `fs`) now provide named exports in ES6 modules. (Gus Caplan) [#20403](#20403) * **timers**: - `timeout.refresh()` has been added to the public API. (Jeremiah Senkpiel) [#20298](#20298) PR-URL: #20724
Notable Changes: * **addons**: - Fixed a memory leak for users of `AsyncResource` and N-API. (Michael Dawson) [#20668](#20668) * **assert**: - The `error` parameter of `assert.throws()` can be an object containing regular expressions now. (Ruben Bridgewater) [#20485](#20485) * **crypto**: - The `authTagLength` option has been made more flexible. (Tobias Nießen) [#20235](#20235), [#20039](#20039) * **http**: - Handling of `close` and `aborted` events has been made more consistent. (Robert Nagy) [#20075](#20075), [#20611](#20611) * Embedder support: - Functions for creating V8 `Isolate` and `Context` objects with Node.js-specific behaviour have been added to the API. (Allen Yonghuang Wang) [#20639](#20639) - Node.js `Environment`s clean up resources before exiting now. (Anna Henningsen) [#19377](#19377) - Support for multi-threaded embedding has been improved. (Anna Henningsen) [#20542](#20542), [#20539](#20539), [#20541](#20541) * **esm**: - Builtin modules (e.g. `fs`) now provide named exports in ES6 modules. (Gus Caplan) [#20403](#20403) * **timers**: - `timeout.refresh()` has been added to the public API. (Jeremiah Senkpiel) [#20298](#20298) PR-URL: #20724
* addons: - Fixed a memory leak for users of `AsyncResource` and N-API. (Michael Dawson) #20668 * assert: - The `error` parameter of `assert.throws()` can be an object containing regular expressions now. (Ruben Bridgewater) #20485 * crypto: - The `authTagLength` option has been made more flexible (Tobias Nießen) #20235) #20039 * esm: - Builtin modules (e.g. `fs`) now provide named exports in ES6 modules. (Gus Caplan) #20403 * http: - Handling of `close` and `aborted` events has been made more consistent. (Robert Nagy) #20075 #20611 * module: - add --preserve-symlinks-main (David Goldstein) #19911 * timers: - `timeout.refresh()` has been added to the public API. (Jeremiah Senkpiel) #20298 * Embedder support: - Functions for creating V8 `Isolate` and `Context` objects with Node.js-specific behaviour have been added to the API. (Allen Yonghuang Wang) #20639 - Node.js `Environment`s clean up resources before exiting now. (Anna Henningsen) #19377 - Support for multi-threaded embedding has been improved. (Anna Henningsen) #20542 #20539 #20541 PR-URL: #20724
* addons: - Fixed a memory leak for users of `AsyncResource` and N-API. (Michael Dawson) #20668 * assert: - The `error` parameter of `assert.throws()` can be an object containing regular expressions now. (Ruben Bridgewater) #20485 * crypto: - The `authTagLength` option has been made more flexible (Tobias Nießen) #20235) #20039 * esm: - Builtin modules (e.g. `fs`) now provide named exports in ES6 modules. (Gus Caplan) #20403 * http: - Handling of `close` and `aborted` events has been made more consistent. (Robert Nagy) #20075 #20611 * module: - add --preserve-symlinks-main (David Goldstein) #19911 * timers: - `timeout.refresh()` has been added to the public API. (Jeremiah Senkpiel) #20298 * Embedder support: - Functions for creating V8 `Isolate` and `Context` objects with Node.js-specific behaviour have been added to the API. (Allen Yonghuang Wang) #20639 - Node.js `Environment`s clean up resources before exiting now. (Anna Henningsen) #19377 - Support for multi-threaded embedding has been improved. (Anna Henningsen) #20542 #20539 #20541 PR-URL: #20724
* addons: - Fixed a memory leak for users of `AsyncResource` and N-API. (Michael Dawson) #20668 * assert: - The `error` parameter of `assert.throws()` can be an object containing regular expressions now. (Ruben Bridgewater) #20485 * crypto: - The `authTagLength` option has been made more flexible (Tobias Nießen) #20235) #20039 * esm: - Builtin modules (e.g. `fs`) now provide named exports in ES6 modules. (Gus Caplan) #20403 * http: - Handling of `close` and `aborted` events has been made more consistent. (Robert Nagy) #20075 #20611 * module: - add --preserve-symlinks-main (David Goldstein) #19911 * timers: - `timeout.refresh()` has been added to the public API. (Jeremiah Senkpiel) #20298 * Embedder support: - Functions for creating V8 `Isolate` and `Context` objects with Node.js-specific behaviour have been added to the API. (Allen Yonghuang Wang) #20639 - Node.js `Environment`s clean up resources before exiting now. (Anna Henningsen) #19377 - Support for multi-threaded embedding has been improved. (Anna Henningsen) #20542 #20539 #20541 PR-URL: #20724
* addons: - Fixed a memory leak for users of `AsyncResource` and N-API. (Michael Dawson) #20668 * assert: - The `error` parameter of `assert.throws()` can be an object containing regular expressions now. (Ruben Bridgewater) #20485 * crypto: - The `authTagLength` option has been made more flexible (Tobias Nießen) #20235) #20039 * esm: - Builtin modules (e.g. `fs`) now provide named exports in ES6 modules. (Gus Caplan) #20403 * http: - Handling of `close` and `aborted` events has been made more consistent. (Robert Nagy) #20075 #20611 * module: - add --preserve-symlinks-main (David Goldstein) #19911 * timers: - `timeout.refresh()` has been added to the public API. (Jeremiah Senkpiel) #20298 * Embedder support: - Functions for creating V8 `Isolate` and `Context` objects with Node.js-specific behaviour have been added to the API. (Allen Yonghuang Wang) #20639 - Node.js `Environment`s clean up resources before exiting now. (Anna Henningsen) #19377 - Support for multi-threaded embedding has been improved. (Anna Henningsen) #20542 #20539 #20541 PR-URL: #20724
This change is used to support addon / embedding scenarios.
An addon/embedding scenario has requirement to create isolate / context with the same setting/tweak that is used in node main thread, instead of duplicating that part of logic by themself, which may lead to conflict in the future.
Please see this commit of napajs as an example.
Checklist
make -j4 test
(UNIX), orvcbuild test
(Windows) passes