[BUG] npm-cli install works from npm repository, but fails from github repository for same package

[getify]

Is there an existing issue for this?

• I have searched the existing issues

This issue exists in the latest npm version

• I am using the latest npm

Current Behavior

If I create an empty folder (like ~/tmp) and inside it issue a npm install caf, my package (published for years on the public npm repository) installs correctly. I can even go into it (~/tmp/node_modules/caf) and run npm install and npm run test:all, and everything runs and tests all pass.

However, if I try the same type of thing (in a clean empty directory!), but issue npm install git+git@github.com:getify/CAF.git to install from the github repo, the installation fails. The main branch (master) of that github repository, at HEAD is the exact same source that the latest npm package release was published from/with, so there shouldn't be any divergence in the files/config/etc.

I can't quite figure out why the install from the git is failing, from the logs of the error that are printed, which I'm finding quite confusing:

...

npm ERR! npm WARN using --force Recommended protections disabled.
npm ERR! npm WARN using --force Recommended protections disabled.
npm ERR! npm WARN using --force Recommended protections disabled.
npm ERR! npm WARN using --force Recommended protections disabled.
npm ERR! npm WARN using --force Recommended protections disabled.
npm ERR! npm WARN using --force Recommended protections disabled.
npm ERR! node:internal/modules/esm/get_format:160
npm ERR!   throw new ERR_UNKNOWN_FILE_EXTENSION(ext, filepath);
npm ERR!         ^
npm ERR!
npm ERR! TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".txt" for /home/getify/.npm/_cacache/tmp/git-cloneXXXXXXwCe8XF/dist/esm/copyright-header.txt
npm ERR!     at Object.getFileProtocolModuleFormat [as file:] (node:internal/modules/esm/get_format:160:9)
npm ERR!     at defaultGetFormat (node:internal/modules/esm/get_format:203:36)
npm ERR!     at defaultLoad (node:internal/modules/esm/load:143:22)
npm ERR!     at async ModuleLoader.load (node:internal/modules/esm/loader:403:7)
npm ERR!     at async ModuleLoader.moduleProvider (node:internal/modules/esm/loader:285:45) {
npm ERR!   code: 'ERR_UNKNOWN_FILE_EXTENSION'
npm ERR! }

...

Full log: 2024-03-24T22_29_12_729Z-debug-0.log

Unfortunately, the temporary location /home/getify/.npm/_cacache/tmp/git-cloneXXXXXXwCe8XF that npm is cloning the git repo into is wiped out when things fail, so I can't go in and inspect that file system to figure out what might be going wrong with what's in there.

Notice that this error message above says Unknown file extension ".txt" for /home/getify/.../dist/esm/copyright-header.txt. What's weird about this is:

1. The copyright-header.txt file only exists under src/, and it's not copied over to dist/esm/ during the build of this package. So I don't know why that file would be in that location as indicated by that error message. It definitely shouldn't be, so something is behaving strangely in a way I have no idea how to understand.

2. Even if there was a .txt file in that folder, what difference should that make to npm?

   This error occurs during the running of the package's test suite. That command is normally run as npm run test:ESM, which is basically TEST_ESM=true node scripts/node-tests.js. THAT test script, when run with the TEST_ESM=true environment variable, spawns a child process to invoke the node-esm-tests.mjs script.

   If you check out that's script's code, from there that test script doesn't do anything unusual. It just imports the ESM version of the library (dist/esm/index.mjs) and runs the qunit test suite. There's nothing in my code that would be trying to parse copyright-header.txt as an ESM module.

   Seems like npm must be doing something different/unexpected here to be... a) getting that .txt file in the wrong location b) trying to parse is as an ESM file.

Expected Behavior

I expect an installation from git and from npm-repository to work the same, if the files are the same (as in this case).

If the test suite is going to run on such an installation (why?!), I expect that environment they're running in to be the same as I run right before I publish to the npm repository, such that if tests pass when publishing, they should pass when installing.

And even if the tests DID fail when installing via git, I wouldn't expect that this should fail and abort the install. The install should complete, and an error message could then say that the test suite didn't complete.

---

Moreover, the only npm lifecycle step (in package.json) I've specified to run the tests is prepublish, which I'm not sure why is being triggered during this git-based installation. I'd honestly rather the tests NOT run in this case, but it's unclear if I have any way to control that with what npm install command I issue.

Steps To Reproduce

1. In a basic linux/posix environment (I'm using debian 11 under WSL2 on windows 11)...
2. Create an empty folder like ~/tmp
3. In this folder, issue npm install git+git@github.com:getify/CAF.git
4. Notice that it will fail while running the package's test suite, against the ESM files, complaining about something like the .txt file extension being unrecognized.

Environment

• npm: 10.5.0
• Node.js: 21.7.1
• OS Name: debian 11 (wsl2 windows 11)
• System Model Name: surface laptop 5
• npm config:

; "user" config from /home/getify/.npmrc

//npm.pkg.github.com:_authToken = (protected)
//registry.npmjs.org/:_authToken = (protected)
email = "redacted"
init-license = "MIT"
init.author = "Kyle Simpson <redacted>"
package-lock = false
save = false
umask = 1

; node bin location = /usr/local/bin/node
; node version = v21.7.1
; npm local prefix = /home/getify/tmp
; npm version = 10.5.0
; cwd = /home/getify/tmp
; HOME = /home/getify
; Run `npm config ls -l` to show all defaults.

[ljharb]

I believe that when you npm install from a github repo, the "prepare" script is run; it's not run when you install from a registry. Could that be the difference here?

[getify]

if you check CAF's package.json, the prepare step only runs the build script, not the test script. prepublish is the only lifecycle step I've defined that explicitly calls my test:all script.

but in either case, I run those build and test scripts myself before publish, and they all pass. and I can run them in an installed copy of the package, and they run fine.

something about how npm is running them is different in a way I haven't anticipated, and don't know how to triage/fix.

[ljharb]

I believe prepublish also runs on git install; you'd want prepublishOnly to have it run only on npm publish.

[getify]

FYI: the main substance of my question here -- not which script runs or not at which time -- is the overall failure and the lack of ability/knowledge to debug it, and that issue remains. Would appreciate any insight or help in debugging.

[revelt]

I can't recreate on Mac. For me, the git works ok, but tests fail at Failed: 'async-generator: onEvent' (2/19):

@getify What if we removed the src/copyright-header.txt and the whole header copyright implementation to pinpoint where is this coming from? (I'm not implying to get rid of copyright banner block permanently, just as an experiment).

Speaking of copyright banners, I do inject the copyright banners on my OS npm packages too. But I do it via esbuild native banner key in the esbuild config, for example, https://github.com/codsen/codsen/blob/main/ops/scripts/esbuild.js#L23 — and it surfaces in all built dist files — very elegant. I see you've got a custom build setup (completely different from what I use), so not sure if esbuild can be a suitable alternative for caf.

PS. thanks for your books, courses and public speaking (and visiting us in London)