`labextension link` does not correct dependencies of linked packages #6109

quigleyj-mavenomics · 2019-03-20T18:23:56Z

This is a rather complex bug, so I created a new repo to make reproduction easier.

https://github.com/quigleyj-mavenomics/jlab-bundler-repro

I have a set of packages being developed in a monorepo (a la Lerna). I have a dependency tree like:

package-a (JupyterLab extension)
  |
  +-> package-b
        |
        +-> package-c

To set them up on JupyterLab, I've linked them:

jupyter labextension link package-c
jupyter labextension link package-b
jupyter labextension link package-a

However, when building package-a, the builder is still hitting the NPM registry to check for package-b and package-c. If these aren't published, the build fails even though B and C are linked.

To Reproduce

The repo I linked above includes a package structure like I've described, and has build instructions. Just clone it, link the packages, and try a build.

Expected behavior

What I'd expect is akin to what yarn link does: Once I've linked a package, the bundler should use that for all in-range version specifiers instead of hitting the registry.

Additional context

There is a workaround which I describe in the repo above. Once the packages are published on a repo, the registry hits will succeed. It doesn't really matter what's on the registry since the linked packages will be used anyway. You can either publish them straight to NPM, or setup a private registry like Verdaccio. With private registries, you can add a line to your yarnrc to make Yarn use it for only a particular scope.

Using the reproduction repo I've setup, this is what the build errors with:

Couldn't find package "@testtest/batman-leaf@0.0.1" required by "@testtest/bar-package@file:linked_packages/testtest-bar-package-0.0.1-26a01b6e57a23185b090ca63cdbb71fc934dec47.tgz" on the "npm" registry.

Separately, it looks like even though this fails the build still attempts to continue, going as far as running webpack (which fails if webpack wasn't installed before the problematic package)

The text was updated successfully, but these errors were encountered:

vidartf · 2019-03-21T10:59:34Z

Would you mind including the full logs (I'm not in a place where I can repro right now)? If you are worried about space concerns, you can make an expandable section with this markdown:

<details><summary>Header text</summary>
<pre>
Lots of
output
go here
</pre>
</details>

Header text

Lots of
output
go here

vidartf · 2019-03-21T12:02:53Z

I'm guessing this is also a duplicate of #5852 ?

quigleyj-mavenomics · 2019-03-21T18:30:37Z

It's distinct from #5852 in that the failure is in a dependency of a local package. I can install package-a just fine, regardless of whether it's public or private. And if package-a only depended on a public package-b, then the build succeeds. It's only when package-a depends on a private package-b that this fails.

Here's the output of the build:

jupyter labextension list

C:\Source\jlab-bundler-repro>jupyter labextension list
JupyterLab v0.35.4
Known labextensions:
   app dir: C:\Users\Joe\AppData\Local\conda\conda\envs\maven-kernel\share\jupyter\lab
        @testtest/foo-package v0.0.1 enabled  ok*
local extensions:

@testtest/foo-package: C:\Source\jlab-bundler-repro\packages\foo
linked packages:

@testtest/bar-package: C:\Source\jlab-bundler-repro\packages\bar

@testtest/batman-leaf: C:\Source\jlab-bundler-repro\packages\batman

jupyter lab build

C:\Source\jlab-bundler-repro>jupyter lab build
[LabBuildApp] JupyterLab 0.35.4
[LabBuildApp] Cleaning C:\Users\Joe\AppData\Local\conda\conda\envs\maven-kernel\share\jupyter\lab
Cleaning C:\Users\Joe\AppData\Local\conda\conda\envs\maven-kernel\share\jupyter\lab...
Success!
[LabBuildApp] Building in C:\Users\Joe\AppData\Local\conda\conda\envs\maven-kernel\share\jupyter\lab
[LabBuildApp] Node v10.15.0
[LabBuildApp] > node c:\source\jupyterlab\jupyterlab\staging\yarn.js install

yarn install v1.9.4

warning package.json: No license field

[1/4] Resolving packages...

success Already up-to-date.

Done in 0.12s.

[LabBuildApp] > C:\Users\Joe\AppData\Roaming\npm\npm.CMD pack C:\Source\jlab-bundler-repro\packages\foo

npm notice

npm notice package: @testtest/foo-package@0.0.1

npm notice === Tarball Contents ===

npm notice 271B package.json

npm notice 283B index.js

npm notice === Tarball Details ===

npm notice name:          @testtest/foo-package

npm notice version:       0.0.1

npm notice filename:      testtest-foo-package-0.0.1.tgz

npm notice package size:  393 B

npm notice unpacked size: 554 B

npm notice shasum:        43199f736543e6c06ec5addbfb3b5ec32f01080e

npm notice integrity:     sha512-8suSHWrwSK/Yc[...]knNvtoqQD7PBw==

npm notice total files:   2

npm notice

testtest-foo-package-0.0.1.tgz

[LabBuildApp] > node c:\source\jupyterlab\jupyterlab\staging\yarn.js install

yarn install v1.9.4

warning package.json: No license field

[1/4] Resolving packages...

success Already up-to-date.

Done in 0.11s.

[LabBuildApp] > C:\Users\Joe\AppData\Roaming\npm\npm.CMD pack C:\Source\jlab-bundler-repro\packages\batman

npm notice

npm notice package: @testtest/batman-leaf@0.0.1

npm notice === Tarball Contents ===

npm notice 144B package.json

npm notice 62B  index.js

npm notice === Tarball Details ===

npm notice name:          @testtest/batman-leaf

npm notice version:       0.0.1

npm notice filename:      testtest-batman-leaf-0.0.1.tgz

npm notice package size:  280 B

npm notice unpacked size: 206 B

npm notice shasum:        a8e5ff11487c125324c82428a44386efee8dd0fc

npm notice integrity:     sha512-vwZ2gBAK/pdlE[...]Gpg4gKjZR/hXw==

npm notice total files:   2

npm notice

testtest-batman-leaf-0.0.1.tgz

[LabBuildApp] > node c:\source\jupyterlab\jupyterlab\staging\yarn.js install

yarn install v1.9.4

warning package.json: No license field

[1/4] Resolving packages...

success Already up-to-date.

Done in 0.11s.

[LabBuildApp] > C:\Users\Joe\AppData\Roaming\npm\npm.CMD pack C:\Source\jlab-bundler-repro\packages\bar

npm notice

npm notice package: @testtest/bar-package@0.0.1

npm notice === Tarball Contents ===

npm notice 164B package.json

npm notice 82B  index.js

npm notice === Tarball Details ===

npm notice name:          @testtest/bar-package

npm notice version:       0.0.1

npm notice filename:      testtest-bar-package-0.0.1.tgz

npm notice package size:  278 B

npm notice unpacked size: 246 B

npm notice shasum:        0d8ffeee180edaace4fd591296424f8d18df1b9d

npm notice integrity:     sha512-SBLDe+X8HcztR[...]uSZ5KfATEi/cg==

npm notice total files:   2

npm notice

testtest-bar-package-0.0.1.tgz

[LabBuildApp] > node c:\source\jupyterlab\jupyterlab\staging\yarn.js install

yarn install v1.9.4

info No lockfile found.

[1/5] Validating package.json...

[2/5] Resolving packages...

error Couldn't find package "@testtest/batman-leaf@0.0.1" required by "@testtest/bar-package@file:linked_packages/testtest-bar-package-0.0.1-26a01b6e57a23185b090ca63cdbb71fc934dec47.tgz" on the "npm" registry.

info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

[LabBuildApp] > node c:\source\jupyterlab\jupyterlab\staging\yarn.js run build

yarn run v1.9.4

$ webpack

'webpack' is not recognized as an internal or external command,

operable program or batch file.

error Command failed with exit code 1.

info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Now if I were to publish the private packages and then try to rebuild, it works as expected. The build succeeds, and it uses my local version of the packages since I linked them. The bug here is that it's still trying to hit the NPM registry during install, when it doesn't need to.

DylanLukes · 2019-08-08T21:09:12Z

I'm also experiencing this issue while attempting to develop an extension.

I am also using a monorepo (with Lerna) and have the dependency structure:

@foo/foo-extension@0.0.1-alpha.1
  |
  +-> @foo/foo@0.0.1-alpha.1

I expect the following to work:

jupyter labextension link ./packages/foo
jupyter labextension install ./packages/foo-extension

But I get:

error Couldn't find package "@foo/foo@^0.0.1-alpha.1" required by "@foo/foo-extension@file:../extensions/foo-foo-extension-0.0.1-alpha.1-b545135ba462af51e473da7bee3e9ebf6f6b5adb.tgz" on the "npm" registry.

yarn linking seems to have no effect whatsoever, and neither does lerna link.

What is the current best workaround/solution?

Somewhat odd addendum, according to the build log, the command where yarn is failing to retrieve the dependency is:

node /Users/dylan/.local/share/virtualenvs/foo-NdSyc0Zw/lib/python3.7/site-packages/jupyterlab/staging/yarn.js install --non-interactive

Running this command on its own works just fine, though it doesn't accomplish anything.

I checked the staging/node_modules directory inside the application directory reported by jupyter lab paths and @foo/foo@0.0.1-alpha.1 is there and checks out... It's absolutely bizarre that jupyter lab build can't find it when it just built it and placed it in the node_modules directory.

quigleyj-mavenomics · 2019-08-08T21:21:16Z

Best workaround we've found so far is to use Verdaccio. We have a local copy running on dev machines where we publish local copies of the monorepo packages. I detailed it a bit more here, about halfway down.

DylanLukes · 2019-08-08T21:25:51Z

Hmmm. The odd thing here is that unlike in your case, my dependency hierarchy is only two levels deep :).

What's happening is that when foo-package is loaded, the dependency on bar-package is correctly transformed to point to the link. However, this process is not recursive: bar-package has a dependency on batman-leaf, but that dependency is not transformed by the build toolchain.

I'm apparently not even having that happen!

DylanLukes · 2019-08-08T21:30:38Z

@quigleyj-mavenomics I've reproduced the issue (on a fork of your reproduction) with only two layers:

https://github.com/DylanLukes/jlab-bundler-repro/tree/only-two-layers

If even this simple case doesn't work (and it doesn't), it would seem jupyter labextension link is completely broken for local-package-based development.

jasongrout · 2019-08-08T21:35:43Z

In ipywidgets, we link all local dependencies in dependency order: https://github.com/jupyter-widgets/ipywidgets/blob/c592184935781d0787bf622bf1659279c4a8b531/dev-install.sh#L55-L58

DylanLukes · 2019-08-08T21:42:22Z

I am also linking all dependencies (there's only one) in dependency order.

The issue here appears to be that jupyter lab build is, for whatever reason, ignoring locally linked dependencies (whether they be yarn linked, jupyter labextension linked, or both) and insisting on looking them up on NPM.

This is despite the entries in $app_dir/staging/package.json's "dependencies" being:

"@foo/foo": "file:linked_packages/foo-foo-0.0.1-alpha.1-bbe8c1617f474d74e3f7a12614fc96971c910920.tgz",
"@foo/foo-extension": "file:../extensions/foo-foo-extension-0.0.1-alpha.1-b545135ba462af51e473da7bee3e9ebf6f6b5adb.tgz"

The relevant yarn.lock lines in the staging directory are:


"@foo/foo-extension@file:../extensions/foo-foo-extension-0.0.1-alpha.1-b545135ba462af51e473da7bee3e9ebf6f6b5adb.tgz":
  version "0.0.1-alpha.1"
  resolved "file:../extensions/foo-foo-extension-0.0.1-alpha.1-b545135ba462af51e473da7bee3e9ebf6f6b5adb.tgz#7c3b38fa2919e13bf73050ecdec4012bf7d238fd"
  dependencies:
    "@foo/foo" "^0.0.1-alpha.1"
    "@jupyterlab/application" "^1.0.2"
    "@jupyterlab/apputils" "^1.0.2"
    "@jupyterlab/coreutils" "^3.0.0"
    "@jupyterlab/notebook" "^1.0.2"
    "@phosphor/coreutils" "^1.3.1"
    "@phosphor/widgets" "^1.8.0"

"@foo/foo@^0.0.1-alpha.1", "@ foo/foo@file:linked_packages/foo-foo-0.0.1-alpha.1-bbe8c1617f474d74e3f7a12614fc96971c910920.tgz":
  version "0.0.1-alpha.1"
  resolved "file:linked_packages/foo-foo-0.0.1-alpha.1-bbe8c1617f474d74e3f7a12614fc96971c910920.tgz#07058fc00a57e18fba07dfdb4b1a305b66cd1ccb"
  dependencies:
    "@jupyterlab/application" "^1.0.2"
    "@jupyterlab/apputils" "^1.0.2"
    "@jupyterlab/console" "^1.0.2"
    "@jupyterlab/coreutils" "^3.0.0"
    "@jupyterlab/notebook" "^1.0.2"
    "@phosphor/coreutils" "^1.3.1"
    "@phosphor/widgets" "^1.8.0"
    react "^16.8.4"

So it doesn't look like yarn is getting it wrong either, it's something specific to jupyter's LabBuildApp.

This happens with both jupyterlab 1.0.0rc and jupyterlab 1.0.2.

jasongrout · 2019-08-08T21:48:50Z

Ah, I notice the @quigleyj-mavenomics mentions in the OP

There is a workaround which I describe in the repo above. Once the packages are published on a repo, the registry hits will succeed. It doesn't really matter what's on the registry since the linked packages will be used anyway.

That would explain it working for ipywidgets - we've published some version of those packages.

DylanLukes · 2019-08-08T21:51:41Z

Yup. I attempted using the verdaccio workaround and can confirm it works. I don't even have to publish updates, as long as the registry lookup succeeds, it uses the locally linked packages as one would expect.

jasongrout · 2019-08-08T21:54:09Z

My guess is that yarn is going out to the registry to check to see if it needs to update the resolved package?

A pointer to a technical experiment: perhaps one of these yarn options is useful in this case:

    --prefer-offline                    use network only if dependencies are not available in local cache
    --pure-lockfile                     don't generate a lockfile
    --frozen-lockfile                   don't generate a lockfile and fail if an update is needed

jasongrout · 2019-08-08T23:20:51Z

Here's how I was able to get it to work with https://github.com/quigleyj-mavenomics/jlab-bundler-repro. The problem is that yarn sees the dependency in, for example, https://github.com/quigleyj-mavenomics/jlab-bundler-repro/blob/fb0cbfc62c253d1f068363ded6325672d9c40d7b/packages/bar/package.json#L5, and notices that it is a traditional dependency (no files, etc.), so it tries to fetch it from the registry in this yarn function. I tried instructing yarn to not reach out to the registry if it could satisfy packages from its local cache, but then the --no-build flags from before had prevented yarn from adding the package to the local cache.

So we have to (a) get yarn to put the linked package in its local cache first, and then (b) tell yarn to use the local cache if it can. We can do (a) by just not doing --no-build, i.e., let yarn build and add the file package to the local cache. For (b), I had to go to the staging directory and directly run the yarn install command with --prefer-offline.

Perhaps we should make --prefer-offline an option, or the default perhaps, if we have local packages? Or perhaps we just switch to instructing people to use a local registry?

SgtPooki · 2019-09-05T19:01:03Z

I got it working with the following steps:

lerna bootstrap --hoist
lerna run build (build all packages)
jupyter labextension link ./packages/project-ui-react-hooks --no-build (ignore error code cause it tries to build anyway)
jupyter labextension install ./packages/project-ui --no-build (ignore error code cause it tries to build anyway)
get jupyter lab paths and CD to the application directory + '/staging'
yarn install --offline
cd back to lerna repo root
jupyter lab build

I was then able to run jupyter lab --watch, and get the jupyterlab server running with my latest extension changes.

blink1073 · 2019-09-05T22:11:29Z

Thanks for following up @SgtPooki! We might be able to change the yarn config itself to fix this. I'll take a look when I am no longer 😷

blink1073 · 2019-09-05T22:20:03Z

@SgtPooki if you add this to the .yarnrc file in staging does rm -rf node_modules && yarn work?

--install.prefer-offline true

SgtPooki · 2019-09-05T23:14:46Z

@blink1073 Almost; adding --install.offline true instead gets it to work.

blink1073 · 2019-09-05T23:19:54Z

Hmm, I'm not sure if offline will work with a clean cache. Can you please try:

yarn cache clean
rm -rf node_modules
yarn

SgtPooki · 2019-09-12T00:12:15Z

Hmm, I'm not sure if offline will work with a clean cache. Can you please try:
yarn cache clean
rm -rf node_modules
yarn

@blink1073 I don't think prefer-offline is an actual option. are you proposing adding that option in the .yarnrc file and then running those commands?

SgtPooki · 2019-09-19T18:43:59Z

and now that I've got multiple unpublished packages that have dependencies on each other, yarn doesn't like this solution anymore. i'm going to try using verdaccio and see if that simplifies things..

kgryte · 2019-10-01T23:05:28Z

@SgtPooki TMK, prefer-offline is an option. See here.

DylanLukes · 2019-10-03T19:27:18Z

@SgtPooki Using verdaccio is definitely a hackish workaround, but I can confirm that it does at least work.

Once you've got it working you don't need to run verdaccio in the background any more, though you may need to start it back up and unlink/re-linkeverything. I haven't figured out exactly what triggers the need to do so, the build process is weirdly flaky.

kgryte · 2019-10-03T19:32:57Z

I can also confirm that using a local registry proxy works.

saulshanabrook · 2019-10-04T13:15:44Z

If anyone needs a workaround for this right now, here is some code to add switch the yarn registry to a local one:

https://github.com/jupyterlab/jupyterlab-data-explorer/pull/93/files#diff-b9cfc7f2cdf78a7f4b91a753d10865a2R14

kgryte · 2019-10-04T18:33:58Z

To add to @saulshanabrook's comment, in order to create a workaround, we had to follow a similar approach to that of @SgtPooki. Notably,

Launch a verdaccio registry.
Publish local (unpublished) packages to verdaccio.
Use Lerna to run various scripts to build the packages (e.g., compile, link, etc).
Attempt to run jupyter lab build.

Often, if running a fresh build after having cleaned Jupyter or in a new environment, we'd encounter a jupyter lab build error complaining about a local package not being found on the npm registry. So, in order to address this, we ran the following steps:

Navigate to the staging directory.
Modify the .yarnrc file in the staging directory to instruct yarn to use the verdaccio registry when attempting to install local unpublished packages.
Explicitly run the local yarn.js file to install extension dependencies.
Navigate back to the project directory.
Attempt to run jupyter lab build.

Typically, this was enough to resolve the build issue relating to unpublished packages. Once the above succeeded, we could use jupyter lab --watch as normal.

saulshanabrook added the tag:Build System label Mar 21, 2019

saulshanabrook added this to the Future milestone Mar 21, 2019

saulshanabrook added the question label Mar 21, 2019

SgtPooki mentioned this issue Sep 5, 2019

jupyterlab-debug-*.log file is outputting the incorrect directory #7153

Open

DylanLukes mentioned this issue Oct 3, 2019

Install a labextension that links a local package #7292

Open

saulshanabrook added tag:Backwards compatible and removed tag:Backwards compatible labels Oct 4, 2019

This was referenced Nov 18, 2019

Created new application package elyra-ai/elyra#208

Merged

Add common Workspace utility class in a common (new) npm project elyra-ai/elyra#148

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`labextension link` does not correct dependencies of linked packages #6109

`labextension link` does not correct dependencies of linked packages #6109

quigleyj-mavenomics commented Mar 20, 2019

vidartf commented Mar 21, 2019

vidartf commented Mar 21, 2019

quigleyj-mavenomics commented Mar 21, 2019

DylanLukes commented Aug 8, 2019 •

edited

quigleyj-mavenomics commented Aug 8, 2019

DylanLukes commented Aug 8, 2019

DylanLukes commented Aug 8, 2019

jasongrout commented Aug 8, 2019 •

edited

DylanLukes commented Aug 8, 2019 •

edited

jasongrout commented Aug 8, 2019

DylanLukes commented Aug 8, 2019

jasongrout commented Aug 8, 2019

jasongrout commented Aug 8, 2019

SgtPooki commented Sep 5, 2019

blink1073 commented Sep 5, 2019 •

edited

blink1073 commented Sep 5, 2019

SgtPooki commented Sep 5, 2019

blink1073 commented Sep 5, 2019

SgtPooki commented Sep 12, 2019

SgtPooki commented Sep 19, 2019

kgryte commented Oct 1, 2019

DylanLukes commented Oct 3, 2019

kgryte commented Oct 3, 2019

saulshanabrook commented Oct 4, 2019

kgryte commented Oct 4, 2019

labextension link does not correct dependencies of linked packages #6109

labextension link does not correct dependencies of linked packages #6109

Comments

quigleyj-mavenomics commented Mar 20, 2019

vidartf commented Mar 21, 2019

vidartf commented Mar 21, 2019

quigleyj-mavenomics commented Mar 21, 2019

DylanLukes commented Aug 8, 2019 • edited

quigleyj-mavenomics commented Aug 8, 2019

DylanLukes commented Aug 8, 2019

DylanLukes commented Aug 8, 2019

jasongrout commented Aug 8, 2019 • edited

DylanLukes commented Aug 8, 2019 • edited

jasongrout commented Aug 8, 2019

DylanLukes commented Aug 8, 2019

jasongrout commented Aug 8, 2019

jasongrout commented Aug 8, 2019

SgtPooki commented Sep 5, 2019

blink1073 commented Sep 5, 2019 • edited

blink1073 commented Sep 5, 2019

SgtPooki commented Sep 5, 2019

blink1073 commented Sep 5, 2019

SgtPooki commented Sep 12, 2019

SgtPooki commented Sep 19, 2019

kgryte commented Oct 1, 2019

DylanLukes commented Oct 3, 2019

kgryte commented Oct 3, 2019

saulshanabrook commented Oct 4, 2019

kgryte commented Oct 4, 2019

`labextension link` does not correct dependencies of linked packages #6109

`labextension link` does not correct dependencies of linked packages #6109

DylanLukes commented Aug 8, 2019 •

edited

jasongrout commented Aug 8, 2019 •

edited

DylanLukes commented Aug 8, 2019 •

edited

blink1073 commented Sep 5, 2019 •

edited