Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: node 20.14.0 #37

Open
wants to merge 23 commits into
base: main
Choose a base branch
from
Open

feat: node 20.14.0 #37

wants to merge 23 commits into from

Conversation

robertsLando
Copy link
Member

@robertsLando robertsLando commented Jun 13, 2024

Seems that latest nodejs 18/20 builds were failing because of old GLIBC versions. I have upgraded both Dockerfile.linux and Dockerfile.linuxcross in order to use gcc 10, this allowed also to remove a workaround added on patches to disable _SYS_RANDOM_H and msign-return-address=all cflag

@robertsLando robertsLando changed the title Node20.14.0 feat: node 20.14.0 Jun 13, 2024
@robertsLando
Copy link
Member Author

@Shogan @sean-stage @Rob3rtS could someone of you give a try to this please?

@robertsLando
Copy link
Member Author

robertsLando commented Jun 13, 2024

@robertsLando
Copy link
Member Author

@axi92 working on this

@robertsLando

This comment was marked as resolved.

@matthias-heller
Copy link

I just tried the patch: The issue is the same as in

#32

node:internal/errors:541
throw error;
^

TypeError [ERR_INVALID_ARG_TYPE]: The "paths[0]" argument must be of type string. Received undefined
at Object.resolve (node:path:171:9)
at resolveMainPath (node:internal/modules/run_main:35:38)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:160:24)
at node:internal/main/run_main_module:28:49 {
code: 'ERR_INVALID_ARG_TYPE'
}

Node.js v20.14.0

When I run the .exe with e.g. myTest.exe app.js then the exe executes the given app.js instead of the internal embedded code when starting myTest.exe only I get the above error

@robertsLando
Copy link
Member Author

@matthias-heller Same for the 18.20.2? Could you test that too please?

@robertsLando
Copy link
Member Author

I have a feel that's something happening only with windows exec as I'm not able to reproduce it with linux

@robertsLando
Copy link
Member Author

@Shogan I know you also got that issue, did you tested the win exe too?

@matthias-heller
Copy link

I have the issue with linux also. 18.20.2 I can't test as the effort in our environment would be quite big

@robertsLando
Copy link
Member Author

@matthias-heller Are you able to provide a minimal script that once packaged reproduces the issue? Because I'm not able to reproduce this

@matthias-heller
Copy link

matthias-heller commented Jul 25, 2024

@robertsLando: Yes seems so. with your binary at least it is build again

node:internal/errors:541
throw error;
^

TypeError [ERR_INVALID_ARG_TYPE]: The "paths[0]" argument must be of type string. Received undefined
at Object.resolve (node:path:171:9)
at resolveMainPath (node:internal/modules/run_main:35:38)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:160:24)
at node:internal/main/run_main_module:28:49 {
code: 'ERR_INVALID_ARG_TYPE'
}

I'll have a look at my build environment as next step first

@matthias-heller
Copy link

@robertsLando: Build issue is fixed and I at least now have the same situation with your binary and my own build one.
The Problem I face now is that I need to put traces into file, as writting to standard out is not a good idea as the fabricator is then not able to work. So I need to put trace information into the file system instead. I'll try to do what you recommended with the node.js patches.
The only thing I see currently after adding to patchProcessObject is this here:

While Building:
patchProcessObject
expandArgv1 = true
argv[0] = C:\Users\MyUser\pkg-cache\v3.5\built-v20.14.0-win-x64

While Running the build binary:
patchProcessObject
expandArgv1 = true
argv[0] = E:\npm\pkg-fetch\pkg-fetch\main.exe
argv[1] = PKG_DUMMY_ENTRYPOINT

Does this help you something?

if (expandArgv1 && process.argv[1] && !StringPrototypeStartsWith(process.argv[1], '-') && process.argv[1] !== 'PKG_DUMMY_ENTRYPOINT') { // Expand process.argv[1] into a full path. log("Expand process.argv[1] into a full path.") const path = require('path'); process.argv.forEach((arg, index) => { log(`argv[${index}] = ${arg}`); }); try { mainEntry = path.resolve(process.argv[1]); process.argv[1] = mainEntry; log("After updating process.argv[1]") process.argv.forEach((arg, index) => { log(`argv[${index}] = ${arg}`); }); } catch { // Continue regardless of error. } }

@matthias-heller
Copy link

One more hint prepareMainThreadExecution returns undefined when running the .exe

@matthias-heller
Copy link

matthias-heller commented Jul 25, 2024

What is a little strange is:

I added some more traces: I run the application and this is what I see

Thu Jul 25 2024 11:07:31 GMT+0200 (GMT+02:00) - enter pkg.js:main
Thu Jul 25 2024 11:07:31 GMT+0200 (GMT+02:00) - enter pkg.js:readPrelude
Thu Jul 25 2024 11:07:31 GMT+0200 (GMT+02:00) - no prelude - remove entrypoint from argv[1]
Thu Jul 25 2024 11:07:31 GMT+0200 (GMT+02:00) - argv[0] = E:\npm\pkg-fetch\pkg-fetch\main.exe
Thu Jul 25 2024 11:07:31 GMT+0200 (GMT+02:00) - leaving pkg.js:main
Thu Jul 25 2024 11:07:31 GMT+0200 (GMT+02:00) - enter pre_execution.js:prepareMainThreadExecution

But when I look into the exe I don't understand why this is happening:

var PAYLOAD_POSITION = '42007552 ' | 0; var PAYLOAD_SIZE = '944 ' | 0; var PRELUDE_POSITION = '42008496 ' | 0; var PRELUDE_SIZE = '74570 ' | 0; if (!PRELUDE_POSITION) { log('no prelude - remove entrypoint from argv[1]'); process.argv.splice(1, 1); process.argv.forEach((arg, index) => { log(`argv[${index}] = ${arg}`); }); return { undoPatch: true }; }

But even with the stuff from the executable the if condition evaluates to true when running the application.

I have another trace when reading the prelude and this is never called

@robertsLando
Copy link
Member Author

robertsLando commented Jul 25, 2024

Build issue is fixed

Could I ask you what it was?

But when I look into the exe I don't understand why this is happening:

Wow that absolutely makes no sense... Could you try to log PRELUDE_POSITION before the if? Also you may change the if to be more explicit like PRELUDE_POSITION == 0

@matthias-heller
Copy link

matthias-heller commented Jul 25, 2024

Build issue was caused as I had a node.js file modified for testing which I reverted back.

It seems the value evaluates to 0 strangely:

Thu Jul 25 2024 11:25:37 GMT+0200 (GMT+02:00) - PAYLOAD_POSITION = 0
Thu Jul 25 2024 11:25:37 GMT+0200 (GMT+02:00) - PAYLOAD_SIZE = 0
Thu Jul 25 2024 11:25:37 GMT+0200 (GMT+02:00) - PRELUDE_POSITION = 0
Thu Jul 25 2024 11:25:37 GMT+0200 (GMT+02:00) - PRELUDE_SIZE = 0

var PAYLOAD_POSITION = '42008064 ' | 0;
var PAYLOAD_SIZE = '944 ' | 0;
var PRELUDE_POSITION = '// PRELUDE_POSITION //' | 0;
var PRELUDE_SIZE = '// PRELUDE_SIZE //' | 0;
log('PAYLOAD_POSITION = ' + PAYLOAD_POSITION);
log('PAYLOAD_SIZE = ' + PAYLOAD_SIZE);
log('PRELUDE_POSITION = ' + PRELUDE_POSITION);
log('PRELUDE_SIZE = ' + PRELUDE_SIZE);

@matthias-heller
Copy link

matthias-heller commented Jul 25, 2024

@robertsLando
It even got more strange:

I changed the code to

var PAYLOAD_POSITION = 0;
var PAYLOAD_POSITION_STRING = '42009088 '.trim();
if(PAYLOAD_POSITION_STRING.length != 0) {
PAYLOAD_POSITION = parseInt(PAYLOAD_POSITION_STRING);
}
var PAYLOAD_SIZE = 0;
var PAYLOAD_SIZE_STRING = '944 '.trim();
if(PAYLOAD_SIZE_STRING.length != 0) {
PAYLOAD_SIZE = parseInt(PAYLOAD_SIZE_STRING);
}

var PRELUDE_POSITION = 0;
var PRELUDE_POSITION_STRING = '42010032              '.trim();
if(PRELUDE_POSITION_STRING.length != 0) {
  PRELUDE_POSITION = parseInt(PRELUDE_POSITION_STRING);
}

var PRELUDE_SIZE = 0;
var PRELUDE_SIZE_STRING = '74570             '.trim();
if(PRELUDE_SIZE_STRING.length != 0) {
  PRELUDE_SIZE = parseInt(PRELUDE_SIZE_STRING);
}

It evaluates to NaN. If I run the code inside a node.js normal environment it does what it should

@matthias-heller
Copy link

@robertsLando: I have the feeling the code is twice inside the final binary.
One time as source code and second time bundled

In the exe I clearly can see

var PAYLOAD_POSITION = 0; var PAYLOAD_POSITION_STRING = '42003456 '.trim(); log('PAYLOAD_POSITION_STRING = ' + PAYLOAD_POSITION_STRING); if(PAYLOAD_POSITION_STRING.length != 0) { PAYLOAD_POSITION = parseInt(PAYLOAD_POSITION_STRING); } var PAYLOAD_SIZE = 0; var PAYLOAD_SIZE_STRING = '944 '.trim(); log('PAYLOAD_SIZE_STRING = ' + PAYLOAD_SIZE_STRING); if(PAYLOAD_SIZE_STRING.length != 0) { PAYLOAD_SIZE = parseInt(PAYLOAD_SIZE_STRING); } var PRELUDE_POSITION = 0; var PRELUDE_POSITION_STRING = '42004400 '.trim(); log('PRELUDE_POSITION_STRING = ' + PRELUDE_POSITION_STRING); if(PRELUDE_POSITION_STRING.length != 0) { PRELUDE_POSITION = parseInt(PRELUDE_POSITION_STRING); } var PRELUDE_SIZE = 0; var PRELUDE_SIZE_STRING = '74570 '.trim(); log('PRELUDE_SIZE_STRING = ' + PRELUDE_SIZE_STRING); if(PRELUDE_SIZE_STRING.length != 0) { PRELUDE_SIZE = parseInt(PRELUDE_SIZE_STRING); } log('PAYLOAD_POSITION = ' + PAYLOAD_POSITION); log('PAYLOAD_SIZE = ' + PAYLOAD_SIZE); log('PRELUDE_POSITION = ' + PRELUDE_POSITION); log('PRELUDE_SIZE = ' + PRELUDE_SIZE);

but when I run the exe I get

Thu Jul 25 2024 12:03:24 GMT+0200 (GMT+02:00) - PAYLOAD_POSITION_STRING = // PAYLOAD_POSITION //
Thu Jul 25 2024 12:03:24 GMT+0200 (GMT+02:00) - PAYLOAD_SIZE_STRING = // PAYLOAD_SIZE //
Thu Jul 25 2024 12:03:24 GMT+0200 (GMT+02:00) - PRELUDE_POSITION_STRING = // PRELUDE_POSITION //
Thu Jul 25 2024 12:03:24 GMT+0200 (GMT+02:00) - PRELUDE_SIZE_STRING = // PRELUDE_SIZE //
Thu Jul 25 2024 12:03:24 GMT+0200 (GMT+02:00) - PAYLOAD_POSITION = NaN
Thu Jul 25 2024 12:03:24 GMT+0200 (GMT+02:00) - PAYLOAD_SIZE = NaN
Thu Jul 25 2024 12:03:24 GMT+0200 (GMT+02:00) - PRELUDE_POSITION = NaN
Thu Jul 25 2024 12:03:24 GMT+0200 (GMT+02:00) - PRELUDE_SIZE = NaN

which is the original inside node\lib\internal\process\pre_execution.js

So this also explains why the original code with the | 0 doesn't work because in the source code it is being replaced but not in the binary code.

That is all what I can probably analyze as I have no idea what happens when bundeling it. Maybe somebody else can take over and have a look why the stuff is twice inside.

@matthias-heller
Copy link

matthias-heller commented Jul 25, 2024

@robertsLando
Yeah but it doesn't happen it seems, I build the binary with pkg and it is replaced in the source code (bundled in the binary) but not when running it. So it has to be a second time bundled with the executable.

The function you are refering to is called. I put a trace also inside

@robertsLando
Copy link
Member Author

@matthias-heller could it be that for some reason the // is someway mis-interpreted during the replace so it fails?

@matthias-heller
Copy link

matthias-heller commented Jul 25, 2024

@robertsLando

I don't think so.

I opened the exe file in a binary editor.
This is what I see in cleartext

(function () {
var require = require;
var fs = require('fs');
var vm = require('vm');
function readPrelude (fd) {
log('enter pkg.js:readPrelude');
var PAYLOAD_POSITION = 0;
var PAYLOAD_POSITION_STRING = '42003456 '.trim();
log('PAYLOAD_POSITION_STRING = ' + PAYLOAD_POSITION_STRING);
if(PAYLOAD_POSITION_STRING.length != 0) {
PAYLOAD_POSITION = parseInt(PAYLOAD_POSITION_STRING);
}
var PAYLOAD_SIZE = 0;
var PAYLOAD_SIZE_STRING = '944 '.trim();
log('PAYLOAD_SIZE_STRING = ' + PAYLOAD_SIZE_STRING);
if(PAYLOAD_SIZE_STRING.length != 0) {
PAYLOAD_SIZE = parseInt(PAYLOAD_SIZE_STRING);
}
var PRELUDE_POSITION = 0;
var PRELUDE_POSITION_STRING = '42004400 '.trim();
log('PRELUDE_POSITION_STRING = ' + PRELUDE_POSITION_STRING);
if(PRELUDE_POSITION_STRING.length != 0) {
PRELUDE_POSITION = parseInt(PRELUDE_POSITION_STRING);
}
var PRELUDE_SIZE = 0;
var PRELUDE_SIZE_STRING = '74570 '.trim();
log('PRELUDE_SIZE_STRING = ' + PRELUDE_SIZE_STRING);
if(PRELUDE_SIZE_STRING.length != 0) {
PRELUDE_SIZE = parseInt(PRELUDE_SIZE_STRING);
}
log('PAYLOAD_POSITION = ' + PAYLOAD_POSITION);
log('PAYLOAD_SIZE = ' + PAYLOAD_SIZE);
log('PRELUDE_POSITION = ' + PRELUDE_POSITION);
log('PRELUDE_SIZE = ' + PRELUDE_SIZE);
if (!PRELUDE_POSITION) {
// no prelude - remove entrypoint from argv[1]
log('no prelude - remove entrypoint from argv[1]');
process.argv.splice(1, 1);
process.argv.forEach((arg, index) => {
log(argv[${index}] = ${arg});
});
return { undoPatch: true };
}
log('prelude - read prelude from file');
var prelude = Buffer.alloc(PRELUDE_SIZE);
var read = fs.readSync(fd, prelude, 0, PRELUDE_SIZE, PRELUDE_POSITION);
if (read !== PRELUDE_SIZE) {
console.error('Pkg: Error reading from file.');
process.exit(1);
}
var s = new vm.Script(prelude, { filename: 'pkg/prelude/bootstrap.js' });
var fn = s.runInThisContext();
log('leaving pkg.js:readPrelude');
return fn(process, require,
console, fd, PAYLOAD_POSITION, PAYLOAD_SIZE);
}
(function () {
log('enter pkg.js:main');
var fd = fs.openSync(process.execPath, 'r');
var result = readPrelude(fd);
if (result && result.undoPatch) {
var bindingFs = process.binding('fs');
fs.internalModuleStat = bindingFs.internalModuleStat;
fs.internalModuleReadJSON = bindingFs.internalModuleReadJSON;
fs.closeSync(fd);
}
log('leaving pkg.js:main');
}());
}());

But when running this executable it is showing the non replaced string

@robertsLando
Copy link
Member Author

I opened the exe file in a binary editor.

Did you opened the builded exe or the patched nodejs exe?

@matthias-heller
Copy link

matthias-heller commented Jul 25, 2024

@robertsLando: I opened the built exe file which I also started

@robertsLando
Copy link
Member Author

@matthias-heller the last thing to test IMO is to try building an exe with a previous nodejs version to see what you see inside it and compare. All this things are very strange

@matthias-heller
Copy link

matthias-heller commented Jul 25, 2024

@robertsLando:
I did the following:
I downloaded the prebuilt binary for node-20.11.1 renamed it to built-20.14.0-win-x64 and run a pkg bundeling.
The final executable is working good.

The stuff which I can compare inside the binary looks same. The byte code content looks totally different. I used the same version of pkg which is 5.12.0

I could see inside the final binary built with node.js 20.14.0 still entries which are the place holders. This is not happening with node.js 20.11.1

screenshot

@robertsLando
Copy link
Member Author

The byte code content looks totally different

This is expected:

Bytecode (reproducibility)

By default, your source code is precompiled to v8 bytecode before being written to the output file. To disable this feature, pass --no-bytecode to pkg.

Why would you want to do this?

If you need a reproducible build process where your executable hashes (e.g. md5, sha1, sha256, etc.) are the same value between builds. Because compiling bytecode is not deterministic (see here or here) it results in executables with differing hashed values. Disabling bytecode compilation allows a given input to always have the same output.

You could try to build it using --no-bytecode so at least you will see the code in clear

@robertsLando
Copy link
Member Author

robertsLando commented Jul 25, 2024

I could see inside the final binary built with node.js 20.14.0 still entries which are the place holders. This is not happening with node.js 20.11.1

Ok I think this is the most important thing now. Could you try to manually replace those entries values on the final binary with the same values you got in the working one?

@matthias-heller
Copy link

matthias-heller commented Jul 25, 2024

@robertsLando: I did, but changing a binary typically won't work, and this is the same situation here. So I tried and the executable crashes down.

@CodeRunsTheWorld
Copy link

CodeRunsTheWorld commented Jul 25, 2024 via email

@matthias-heller
Copy link

matthias-heller commented Jul 25, 2024

Yes I did. But this is inside the bytecode section I can't quarantee that the stuff before the string which is bytecode is still okay. But even without bytecode it crashes as then I will change the filesize anyway.
Also I think doing is would just show that it works, but the root cause has to be found and solved.

So at least we know what the problem is. The four variables which should be filled while building the binary evaluate to 0 and it is not a matter of '// PRELUDE_SIZE //' | 0 but realle PRELUDE_SIZE and all the other variables are missing

@robertsLando
Copy link
Member Author

robertsLando commented Jul 25, 2024

I confirm that also on linux build that PRELUDE_SIZE is not present, this means that for some reason the injectPlaceholder fails

https://github.com/yao-pkg/pkg/blob/ba407efef1d2e611d4c192b93295dcaa023fea79/lib/producer.ts#L51

If so it meanst the inject should throw here:

https://github.com/yao-pkg/pkg/blob/ba407efef1d2e611d4c192b93295dcaa023fea79/lib/producer.ts#L62

@matthias-heller
Copy link

@robertsLando: Thank you for validating under linux. Are the other three place holders missing as well in linux?

@robertsLando
Copy link
Member Author

@robertsLando: Thank you for validating under linux. Are the other three place holders missing as well in linux?

Yep

@matthias-heller
Copy link

Okay great at least we found the problem. So my roughly two days of investigating in my freetime was not waste. Now we have to solve it

@robertsLando
Copy link
Member Author

Yeah and thanks for that 🙏🏼 I wasted the same time making the other archs working 😆 Maintaining this repo is just a pain but unfortunately there is no real alternative right now as node SEA doesn't suite well for complex applications (yet).

Anyway even if we discovered that I stll have no clue why that happens only on windows :(

@matthias-heller
Copy link

Yeah and thanks for that 🙏🏼 I wasted the same time making the other archs working 😆 Maintaining this repo is just a pain but unfortunately there is no real alternative right now as node SEA doesn't suite well for complex applications (yet).

Anyway even if we discovered that I stll have no clue why that happens only on windows :(

Yeah, but I thought under linux the place holders are also not filled, so maybe you can have a look why this happens and when it is somehow solved, I can try it under windows as well. If I could offer any help let me know.

@marcus-j-davies
Copy link

marcus-j-davies commented Jul 25, 2024

Just a stab in the dark here (and I am more then aware I could be looking like a twit/cowboy)

LIne endings.... between *nix and windows, could that be messing with buffer lengths somehow, maybe in 20.14 they treated this differently ?

given the difference in length between \n and \r\n

could it be pkg interprets these differently? - at least something in 20.14 causing it todo so

@matthias-heller
Copy link

Any news on this topic?

@robertsLando
Copy link
Member Author

@matthias-heller Unfortunately nope, I was thinking about merging this and do a new release then wait for people asking why windows isn't working to ask them help 😆 I know it's not cool but I'm running out of ideas and time is limited

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants