Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Lost support submodules #2774

Open
trofim24 opened this issue Feb 25, 2021 · 11 comments
Open

[BUG] Lost support submodules #2774

trofim24 opened this issue Feb 25, 2021 · 11 comments
Labels
Bug thing that needs fixing Priority 1 high priority issue Release 7.x work is associated with a specific npm 7 release Release 8.x work is associated with a specific npm 8 release Release 9.x work is associated with a specific npm 9 release Release 10.x

Comments

@trofim24
Copy link

Current Behavior:

npm version 7 (I tried both 7.0.0 and the last 7.5.6) does not download submodules.

Expected Behavior:

npm version 6.14.11 has support submodules.

Steps To Reproduce:

I created 3 repos to reproduce the problem. The first contains package.json with dependencies to install. The second contains submodule. The third contains only Readme.md

  1. git clone https://github.com/trofim24/npm.git
  2. npm install
    If you do this on version 6.14.11 then the node_modules/npm-bug/npm-bug-submodule folder will exist.
    If you do this on version 7.5.6 then the node_modules/npm-bug/npm-bug-submodule folder will not exist.

Environment:

  • OS: Windows 10 2004 or Debian GNU/Linux 9.13
  • npm: 6.14.11 and npm: 7.5.6
@trofim24 trofim24 added Bug thing that needs fixing Needs Triage needs review for next steps Release 7.x work is associated with a specific npm 7 release labels Feb 25, 2021
@ShockwaveNN ShockwaveNN mentioned this issue Feb 26, 2021
Merged
@ShockwaveNN ShockwaveNN mentioned this issue Mar 15, 2021
Closed
@trofim24
Copy link
Author

Any updates?

@zpv
Copy link

zpv commented Mar 26, 2021

Also experiencing this issue and breaks my workflow. I reverted back to 6 in the meantime.
Related (old) issue: npm/npm#6700

@noerw noerw mentioned this issue Apr 7, 2021
Merged
@darcyclarke darcyclarke added Priority 2 secondary priority issue and removed Needs Triage needs review for next steps labels Apr 9, 2021
@hydai hydai mentioned this issue Oct 20, 2021
Merged
@martenrichter martenrichter mentioned this issue Mar 8, 2022
Closed
@darcyclarke darcyclarke added Priority 1 high priority issue and removed Priority 2 secondary priority issue labels Mar 28, 2022
cztomsik added a commit to cztomsik/graffiti that referenced this issue Oct 15, 2022
@cztomsik
fix github install (npm/cli#2774)
f982e21 
npm workaround - it's documented but it doesn't work
@trofim24
Copy link
Author

Any updates?

@ljharb ljharb added Release 8.x work is associated with a specific npm 8 release Release 9.x work is associated with a specific npm 9 release labels Feb 13, 2023
@Mhmonicox Mhmonicox mentioned this issue Feb 15, 2023
Open
@externl externl mentioned this issue Feb 21, 2023
Open
domq pushed a commit to epfl-si/zeebe-db-monitor that referenced this issue Feb 28, 2023
[fix] Downgrade npm for Git submodule support
d1ca562 
`npm install` used to `git clone --recursive` the things that it builds out of GitHub; but it [doesn't do that anymore](npm/cli#2774). Unfortunately, we depend on that feature, because a submodule is how https://github.com/Level/rocksdb pulls in the Facebook code.

This fixes a really absconse error message, which is a huge pain to debug BTW:

```
npm ERR! make: *** No rule to make target 'Release/obj.target/rocksdb/deps/rocksdb/rocksdb/cache/cache.o', needed by 'Release/obj.target/deps/rocksdb/rocksdb.a'.  Stop.
```

(That's because there's no source code under `deps/rocksdb/rocksdb`, on account of that directory being a Git submodule.)

It seems... difficult to convince npm to keep its temporary files for inspection. I had to temporarily push a hacked `package.json` into the `master` branch of `epfl-si/rocksdb-node`, with an `install` script entry that reads

```
    "install": "node-gyp-build || (sleep 3600; exit 1)",
```

Confusingly enough, modern `npm`s will do the right thing (i.e. pull the submodules) when one runs e.g. `npm i -g epfl-si/rocksdb-node`; but not when said `epfl-si/rocksdb-node` is a dependency in a `package.json`, as is the case here. Go figure.
@domq
Copy link

domq commented Feb 28, 2023

Curiously enough, using the latest version of npm, (e.g.) npm i -g https://github.com/Level/rocksdb does what we want i.e. pull the deps/rocksdb/rocksdb submodule out of Facebook. But not when "github.com/Level/rocksdb" is in a package.json as a dependency.

@ljharb
Copy link
Collaborator

ljharb commented Feb 28, 2023

@domq presumably npm install https://github.com/Level/rocksdb would match package.json (global installs are different than local ones)

@domq
Copy link

domq commented Feb 28, 2023

@ljharb I'm not sure what you mean with “match package.json”, in my experiments it's the git behavior (i.e. whether the clone is recursive or not), that differs between the two situations.

@ljharb
Copy link
Collaborator

ljharb commented Feb 28, 2023

@domq sorry, what i mean is, you said npm install -g does what you want, but npm install (when it's in package.json) does not. I was asking about what npm install https://github.com/Level/rocksdb does, which is a third, potentially different thing.

@domq
Copy link

domq commented Feb 28, 2023 •
edited

@ljharb oh, I get it now, sorry. The behavior of npm install https://github.com/Level/rocksdb is indeed the same as when the dependency is in a package.json as you guessed, i.e. a nonrecursive clone is used for the build.

domq pushed a commit to epfl-si/zeebe-db-monitor that referenced this issue Feb 28, 2023
[fix] Downgrade npm for Git submodule support
b03e833 
`npm install` used to `git clone --recursive` the things that it builds out of GitHub; but it [doesn't do that anymore](npm/cli#2774). Unfortunately, we depend on that feature, because a submodule is how https://github.com/Level/rocksdb pulls in the Facebook code.

This fixes a really absconse error message, which is a huge pain to debug BTW:

```
npm ERR! make: *** No rule to make target 'Release/obj.target/rocksdb/deps/rocksdb/rocksdb/cache/cache.o', needed by 'Release/obj.target/deps/rocksdb/rocksdb.a'.  Stop.
```

(That's because there's no source code under `deps/rocksdb/rocksdb`, on account of that directory being a Git submodule.)

It seems... difficult to convince npm to keep its temporary files for inspection. I had to temporarily push a hacked `package.json` into the `master` branch of `epfl-si/rocksdb-node`, with an `install` script entry that reads

```
    "install": "node-gyp-build || (sleep 3600; exit 1)",
```

Confusingly enough, modern `npm`s will do the right thing (i.e. pull the submodules) when one runs e.g. `npm i -g epfl-si/rocksdb-node`; but not when said `epfl-si/rocksdb-node` is a dependency in a `package.json`, as is the case here. Go figure.
domq pushed a commit to epfl-si/zeebe-db-monitor that referenced this issue Mar 1, 2023
[fix] Downgrade npm for Git submodule support
eee83e8 
`npm install` used to `git clone --recursive` the things that it builds out of GitHub; but it [doesn't do that anymore](npm/cli#2774). Unfortunately, we depend on that feature, because a submodule is how https://github.com/Level/rocksdb pulls in the Facebook code.

This fixes a really absconse error message, which is a huge pain to debug BTW:

```
npm ERR! make: *** No rule to make target 'Release/obj.target/rocksdb/deps/rocksdb/rocksdb/cache/cache.o', needed by 'Release/obj.target/deps/rocksdb/rocksdb.a'.  Stop.
```

(That's because there's no source code under `deps/rocksdb/rocksdb`, on account of that directory being a Git submodule.)

It seems... difficult to convince npm to keep its temporary files for inspection. I had to temporarily push a hacked `package.json` into the `master` branch of `epfl-si/rocksdb-node`, with an `install` script entry that reads

```
    "install": "node-gyp-build || (sleep 3600; exit 1)",
```

Confusingly enough, modern `npm`s will do the right thing (i.e. pull the submodules) when one runs e.g. `npm i -g epfl-si/rocksdb-node`; but not when said `epfl-si/rocksdb-node` is a dependency in a `package.json`, as is the case here. Go figure.
domq pushed a commit to epfl-si/zeebe-db-monitor that referenced this issue Mar 1, 2023
[fork] the rocksdb Node module
519527c 
💡 Because of npm/cli#2774 , this means that we must now use version 6 of npm to rebuild (!), like so:

```
npx npm@6 install
``
domq pushed a commit to epfl-si/zeebe-db-monitor that referenced this issue Mar 1, 2023
[fix] Downgrade npm for Git submodule support
432dfd0 
`npm install` used to `git clone --recursive` the things that it builds out of GitHub; but it [doesn't do that anymore](npm/cli#2774). Unfortunately, we depend on that feature, because a submodule is how https://github.com/Level/rocksdb pulls in the Facebook code.

This fixes a really absconse error message, which is a huge pain to debug BTW:

```
npm ERR! make: *** No rule to make target 'Release/obj.target/rocksdb/deps/rocksdb/rocksdb/cache/cache.o', needed by 'Release/obj.target/deps/rocksdb/rocksdb.a'.  Stop.
```

(That's because there's no source code under `deps/rocksdb/rocksdb`, on account of that directory being a Git submodule.)

It seems... difficult to convince npm to keep its temporary files for inspection. I had to temporarily push a hacked `package.json` into the `master` branch of `epfl-si/rocksdb-node`, with an `install` script entry that reads

```
    "install": "node-gyp-build || (sleep 3600; exit 1)",
```

Confusingly enough, modern `npm`s will do the right thing (i.e. pull the submodules) when one runs e.g. `npm i -g epfl-si/rocksdb-node`; but not when said `epfl-si/rocksdb-node` is a dependency in a `package.json`, as is the case here. Go figure.
domq pushed a commit to epfl-si/zeebe-db-monitor that referenced this issue Mar 1, 2023
[fork] the rocksdb Node module
9289a5a 
💡 Because of npm/cli#2774 , this means that we must now use version 6 of npm to rebuild (!), like so:

```
npx npm@6 install
``
domq pushed a commit to epfl-si/zeebe-db-monitor that referenced this issue Mar 1, 2023
[fix] Downgrade npm for Git submodule support
974e7d9 
`npm install` used to `git clone --recursive` the things that it builds out of GitHub; but it [doesn't do that anymore](npm/cli#2774). Unfortunately, we depend on that feature, because a submodule is how https://github.com/Level/rocksdb pulls in the Facebook code.

This fixes a really absconse error message, which is a huge pain to debug BTW:

```
npm ERR! make: *** No rule to make target 'Release/obj.target/rocksdb/deps/rocksdb/rocksdb/cache/cache.o', needed by 'Release/obj.target/deps/rocksdb/rocksdb.a'.  Stop.
```

(That's because there's no source code under `deps/rocksdb/rocksdb`, on account of that directory being a Git submodule.)

It seems... difficult to convince npm to keep its temporary files for inspection. I had to temporarily push a hacked `package.json` into the `master` branch of `epfl-si/rocksdb-node`, with an `install` script entry that reads

```
    "install": "node-gyp-build || (sleep 3600; exit 1)",
```

Confusingly enough, modern `npm`s will do the right thing (i.e. pull the submodules) when one runs e.g. `npm i -g epfl-si/rocksdb-node`; but not when said `epfl-si/rocksdb-node` is a dependency in a `package.json`, as is the case here. Go figure.
@cjbj
Copy link

cjbj commented May 25, 2023

Still not fixed in npm 9.6.7

@jackromo888 jackromo888 mentioned this issue Jun 27, 2023
Open
@Mary12jo33 Mary12jo33 mentioned this issue Jun 29, 2023
Open
@NOUIY NOUIY mentioned this issue Jun 29, 2023
Open
@DeeDeeG
Copy link

DeeDeeG commented Oct 24, 2023

This is both a bug and a documentation issue. I can confirm this bug (the loss of submodule/recursive cloning in dependencies) is still true in npm 10.

The documentation claims submodules will be cloned, but submodules are actually not cloned anymore.

If the repository makes use of submodules, those submodules will be cloned as well.

- From: https://docs.npmjs.com/cli/v10/commands/npm-install

Test Results:

  • Submodules are cloned (as in, it works as described in the npm install documentation) if you use npm 6 (npm v6.14.18). ✅
  • Submodule cloning doesn't happen for (as in, docs are inaccurate for) npm 7 (v7.24.2), npm 8 (v8.19.4), npm 9 (v9.9.0), and npm latest/10 (v10.2.1). ❌

Implications of the bug:
This loss of submodule/recursive cloning breaks some packages when used as git dependencies. For example, Atom's superstring library doesn't build properly on Windows with npm 7.x or newer, as a git dependency, due to a missing vendored win-iconv build-time dependency, which is vendored in as a submodule under the superstring repo. This was working as of npm 6.

What to do about it?:
I'd love for submodule/recursive cloning for git dependencies to come back in npm latest. It would un-break some dependencies we still rely on. But if this support is not coming back any time soon, I feel like the docs should be updated in the mean-time to not claim submodules will be cloned when they actually won't. Thanks for considering.

@DeeDeeG DeeDeeG mentioned this issue Oct 24, 2023
Merged
@NOUIY NOUIY mentioned this issue Dec 2, 2023
Open
@CALISOULB CALISOULB mentioned this issue Dec 20, 2023
Open
@siemhesda siemhesda mentioned this issue Jan 18, 2024
Closed
@ljharb
Copy link
Collaborator

ljharb commented Jan 18, 2024

fwiw, I suddenly have this need as well on two different repos - i'm using a submodule to pull in test fixtures, and having npm install ensure the submodules are available would be very helpful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug thing that needs fixing Priority 1 high priority issue Release 7.x work is associated with a specific npm 7 release Release 8.x work is associated with a specific npm 8 release Release 9.x work is associated with a specific npm 9 release Release 10.x
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@ljharb @cjbj @darcyclarke @domq @trofim24 @zpv @DeeDeeG