Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: enable loading internal modules from disk #31321

Open
wants to merge 1 commit into
base: master
from

Conversation

@devsnek
Copy link
Member

devsnek commented Jan 11, 2020

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • commit message follows commit guidelines
@devsnek devsnek requested a review from joyeecheung Jan 11, 2020
@nodejs-github-bot

This comment has been minimized.

Copy link
Member

addaleax left a comment

Still sad that this isn’t a runtime option, but thanks for picking this up again.

src/node_native_module.cc Outdated Show resolved Hide resolved
@devsnek

This comment has been minimized.

Copy link
Member Author

devsnek commented Jan 12, 2020

@addaleax I'd be happy to make it a runtime option. I'm not sure how I'd get data from the options parser to the internal module loader though.

@tniessen

This comment has been minimized.

Copy link
Member

tniessen commented Jan 12, 2020

This is great! I am fine with it being a configure option (maybe just for now), given that we probably wouldn't want people to use it outside of development. (For example, the current PR seems to resolve the path relative to the current working directory, which seems dangerous outside of the repo.)

src/node_native_module.cc Outdated Show resolved Hide resolved
src/node_native_module.cc Outdated Show resolved Hide resolved
src/node_native_module.cc Outdated Show resolved Hide resolved
configure.py Outdated Show resolved Hide resolved
@Trott

This comment has been minimized.

Copy link
Member

Trott commented Jan 13, 2020

Should this print a warning telling end users not to use it?

@jasnell

This comment has been minimized.

Copy link
Member

jasnell commented Jan 13, 2020

+1 to emitting a warning on this.

@addaleax

This comment has been minimized.

Copy link
Member

addaleax commented Jan 13, 2020

I think if we turn this into a runtime option, the warning may be appropriate, but for a compile time option I don’t quite see the point – whoever built the Node.js binary explicitly opted into a developer feature, and adding the warning is going to make running the test suite infeasible, which makes it less useful as a developer feature.

@targos

This comment has been minimized.

Copy link
Member

targos commented Jan 13, 2020

Maybe James was thinking about a build time warning?

@jasnell

This comment has been minimized.

Copy link
Member

jasnell commented Jan 13, 2020

Maybe James was thinking about a build time warning?

Yes, thank you. That's what I meant.

Regarding the idea of a runtime option: I see the benefit but I also see significant risk. The fact that we have to expose --expose-internals is bad enough. I'd prefer we start with the configuration option and go from there.

Copy link
Member

joyeecheung left a comment

A couple of non-blocking nits. Happy to see this happening!

I think it's possible to add a test for it (at least by testing that adding a new module works), to feature-detect I think it makes sense to just use process.features. But that can be added later since this is just a developer feature after all.

src/node_native_module.cc Outdated Show resolved Hide resolved
configure.py Outdated Show resolved Hide resolved
@devsnek

This comment has been minimized.

Copy link
Member Author

devsnek commented Jan 14, 2020

I'm not sure how to test this cc @nodejs/build

@richardlau

This comment has been minimized.

Copy link
Member

richardlau commented Jan 14, 2020

I'm not sure how to test this cc @nodejs/build

If you want to do a one-off build you can run https://ci.nodejs.org/job/node-test-commit-custom-suites-freestyle/ with custom args passed to configure.
image

@devsnek

This comment has been minimized.

Copy link
Member Author

devsnek commented Jan 14, 2020

@richardlau i mean continuously. I guess we'd need to add a new box/configuration?

@devsnek devsnek force-pushed the devsnek:load-internal-modules-from-disk branch from 52a14a6 to c44133d Jan 14, 2020
configure.py Outdated Show resolved Hide resolved
@richardlau

This comment has been minimized.

Copy link
Member

richardlau commented Jan 14, 2020

@richardlau i mean continuously. I guess we'd need to add a new box/configuration?

You probably wouldn't need a new box but probably would need a new job since you'd have to build Node.js with different options passed to configure. The nearest to that we currently have are the various sharedlibs jobs but we do not test every option that can be passed to configure.

The Build WG is currently very stretched though so there may not be much enthusiasm for adding another build configuration for a developer feature.

@richardlau

This comment has been minimized.

Copy link
Member

richardlau commented Jan 14, 2020

re. new build configurations nodejs/build#1982 / #30057 would be ideal for this sort of thing but unfortunately it's stuck (nodejs/build#1982 (comment)).

@devsnek devsnek force-pushed the devsnek:load-internal-modules-from-disk branch 2 times, most recently from 6ecfc90 to dcc29cb Jan 14, 2020
@devsnek

This comment has been minimized.

Copy link
Member Author

devsnek commented Jan 14, 2020

cctest failures when this is enabled:

[----------] 6 tests from BaseObjectPtrTest
[ RUN      ] BaseObjectPtrTest.ScopedDetached
../test/cctest/test_base_object_ptr.cc:50: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 1
  0
../test/cctest/test_base_object_ptr.cc:53: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 2
  1
../test/cctest/test_base_object_ptr.cc:55: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 1
  0
[  FAILED  ] BaseObjectPtrTest.ScopedDetached (33 ms)
[ RUN      ] BaseObjectPtrTest.ScopedDetachedWithWeak
../test/cctest/test_base_object_ptr.cc:66: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 1
  0
../test/cctest/test_base_object_ptr.cc:70: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 2
  1
../test/cctest/test_base_object_ptr.cc:73: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 1
  0
[  FAILED  ] BaseObjectPtrTest.ScopedDetachedWithWeak (21 ms)
[ RUN      ] BaseObjectPtrTest.Undetached
../test/cctest/test_base_object_ptr.cc:87: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 2
  1
../test/cctest/test_base_object_ptr.cc:83: Failure
Expected equality of these values:
  static_cast<Environment*>(arg)->base_object_count()
    Which is: 1
  0
[  FAILED  ] BaseObjectPtrTest.Undetached (20 ms)
[ RUN      ] BaseObjectPtrTest.GCWeak
../test/cctest/test_base_object_ptr.cc:104: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 2
  1
../test/cctest/test_base_object_ptr.cc:111: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 2
  1
[  FAILED  ] BaseObjectPtrTest.GCWeak (20 ms)
[ RUN      ] BaseObjectPtrTest.Moveable
../test/cctest/test_base_object_ptr.cc:129: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 2
  1
../test/cctest/test_base_object_ptr.cc:140: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 2
  1
../test/cctest/test_base_object_ptr.cc:145: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 1
  0
[  FAILED  ] BaseObjectPtrTest.Moveable (30 ms)
[ RUN      ] BaseObjectPtrTest.NestedClasses
../test/cctest/test_base_object_ptr.cc:175: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 4
  3
../test/cctest/test_base_object_ptr.cc:167: Failure
Expected equality of these values:
  static_cast<Environment*>(arg)->base_object_count()
    Which is: 1
  0
[  FAILED  ] BaseObjectPtrTest.NestedClasses (19 ms)
[----------] 6 tests from BaseObjectPtrTest (143 ms total)
@rvagg

This comment has been minimized.

Copy link
Member

rvagg commented Jan 14, 2020

I don't mind putting in a bit of work to extend the sharedlibs set to include new build configurations if the build configuration is above some poorly defined level of significance. Basically, is this anticipated to be a commonly used build option? Do we think this is going to grow into a more significant thing? If it's a key feature then I could do some work with some pointers on how to compile and run the tests with this option unless this is just some obscure thing in the corner that serves the use of a couple of people with special needs.

@devsnek

This comment has been minimized.

Copy link
Member Author

devsnek commented Jan 14, 2020

I'd imagine people working on node core would just run with this option enabled (unless they were working specifically on the internal module loader itself)

I don't think it's a huge priority to get testing infra for it though.

@rvagg

This comment has been minimized.

Copy link
Member

rvagg commented Jan 14, 2020

ok, so maybe just folks in here should remain aware that it's not tested and if this feature becomes something with wider deployment than core folks then someone should open an issue with build to get it tested.

@rvagg rvagg closed this Jan 14, 2020
@rvagg rvagg reopened this Jan 14, 2020
@rvagg

This comment has been minimized.

Copy link
Member

rvagg commented Jan 14, 2020

(missed the button, sorry)

@mmarchini

This comment has been minimized.

Copy link
Member

mmarchini commented Jan 14, 2020

Can this be tested on GH Actions instead? It would give some coverage and should be lower maintenance than doing it on Jenkins right now.

@Trott
Trott approved these changes Jan 15, 2020
src/node_native_module.cc Outdated Show resolved Hide resolved
src/node_native_module.cc Show resolved Hide resolved
@devsnek devsnek force-pushed the devsnek:load-internal-modules-from-disk branch from dcc29cb to fb0f519 Jan 16, 2020
@nodejs-github-bot

This comment has been minimized.

@nodejs-github-bot

This comment has been minimized.

@devsnek

This comment has been minimized.

Copy link
Member Author

devsnek commented Jan 17, 2020

this is not ready to land, see #31321 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.