Prevent Yarn from rewriting the lockfile when switching between online ⇄ offline #51

arcanis · 2017-03-07T00:35:39Z

First draft cc @bestander

cpojer · 2017-03-07T08:12:44Z

This is great!

bestander · 2017-03-07T12:46:56Z

0000-offline-resolution-field.md

+
+Yarn currently has two different type of values for the lockfile `resolved` fields:
+
+  - When online, they're in the form `${source}/${name}-${version}.tar.gz#${hash}`


also it can be a git URL e.g. ssh://git@bitbucket.org/bestander/test-repo.git, a link: or a file:.

bestander · 2017-03-07T12:53:07Z

0000-offline-resolution-field.md

+
+    - **Rational: Why removing the source part of the resolved field?**
+
+      The source is redundant, and misleading:


Another advantage is that Yarn can start downloading the dependencies immediately skipping the resolution step.
Also there may be some caveats regarding proxies and private packages.

Also there may be some caveats regarding proxies and private packages.

It might also causes issue on this regard even with the source; for example, a dev might use an internal IP as mirror, and a public URL on CI. Baking the source in the lockfile means that this cannot be done without postprocessing the lockfile (since the CI won't have access to this URL). Removing the source would help resolve this issue, since then the CI would use whatever registry it is configured to use rather than the ones specified in the lockfile.

Agree but it is a separate issue/feature

bestander · 2017-03-07T13:14:46Z

I wonder if getting rid of domain from URL and making yarn.lock not changed when using Offline Mirror are two different problems you are trying to tackle.
Would it be simpler to do deal with them separately?

arcanis · 2017-03-07T15:43:52Z

A first iteration could uniformize the two processes by baking the source into both offline & online package resolutions. There might be a bit less code involved, but I'm not completely sure (because there wouldn't be any distinction between offline and online, we would have to check the offline cache anyway to see if a matching file exists). The only step that would be avoided would be to dynamically resolve the source against the package.json (because then we would just use whatever source is set in the lockfile instead).

bestander · 2017-03-07T16:00:25Z

Can you give an example how yarn.lock will look?

arcanis · 2017-03-07T17:46:52Z

If we strip the source: (note that even the git repo is only resolved to its name + hash)

"@manaflair/term-strings@https://github.com/manaflair/term-strings.git#0782bd21a953f29f6425412865b826a62a06722c":
  version "0.7.0"
  resolved "term-strings-0782bd21a953f29f6425412865b826a62a06722c.tar.gz#0782bd21a953f29f6425412865b826a62a06722c"
  dependencies:
    babel-cli "^6.18.0"
    color-diff "^1.0.0"
    core-js "^2.4.1"

abbrev@1:
  version "1.1.0"
  resolved "abbrev-1.1.0-d0554c2256636e2f56e7c2e5ad183f859428d81f.tgz#d0554c2256636e2f56e7c2e5ad183f859428d81f"

ajv@^4.9.1:
  version "4.11.4"
  resolved "ajv-4.11.4-ebf3a55d4b132ea60ff5847ae85d2ef069960b45.tgz#ebf3a55d4b132ea60ff5847ae85d2ef069960b45"
  dependencies:
    co "^4.6.0"
    json-stable-stringify "^1.0.1"

If we forcefully write the source even when offline:

"@manaflair/term-strings@https://github.com/manaflair/term-strings.git#0782bd21a953f29f6425412865b826a62a06722c":
  version "0.7.0"
  resolved "https://github.com/manaflair/term-strings.git#0782bd21a953f29f6425412865b826a62a06722c"
  dependencies:
    babel-cli "^6.18.0"
    color-diff "^1.0.0"
    core-js "^2.4.1"

abbrev@1:
  version "1.1.0"
  resolved "https://registry.yarnpkg.com/abbrev/-/abbrev-1.1.0.tgz#d0554c2256636e2f56e7c2e5ad183f859428d81f"

ajv@^4.9.1:
  version "4.11.4"
  resolved "https://registry.yarnpkg.com/ajv/-/ajv-4.11.4.tgz#ebf3a55d4b132ea60ff5847ae85d2ef069960b45"
  dependencies:
    co "^4.6.0"
    json-stable-stringify "^1.0.1"

bestander

I think these are all good ideas that should be evaluated separately.
Let's also focus on speed and backwards compatibility.

bestander · 2017-03-11T20:18:27Z

0000-offline-resolution-field.md

+
+    - **Rational: Why removing the source part of the resolved field?**
+
+      The source is redundant, and misleading:


Agree but it is a separate issue/feature

bestander · 2017-03-11T20:21:02Z

0000-offline-resolution-field.md

+
+        - If we change our registry URL, the package itself will not have changed. The lockfile should not be modified.
+
+        - The source is already in the package.json file - we can use it when we need to, since we will have checked by then that the resolved version still match (when working with a registry). Worst case scenario (direct link) we will catch that the file will have changed when checking the hash - but that's not something we could have avoided with storing the source in the lockfile anyway.


The source at package.json is loose that is resolved to a fixed URL in package-resolver.
Yarn gains speed by caching and freezing resolution step in the lockfile

bestander · 2017-03-11T20:22:10Z

0000-offline-resolution-field.md

+
+        - The source is already in the package.json file - we can use it when we need to, since we will have checked by then that the resolved version still match (when working with a registry). Worst case scenario (direct link) we will catch that the file will have changed when checking the hash - but that's not something we could have avoided with storing the source in the lockfile anyway.
+
+  - (Optional) Splitting the resulting `resolved` field into two others: `hash` (the hash of the tarballs) and `tarball` (the tarball name).


That might be useful especially that we want to migrate to sha2/sha3

bestander · 2017-03-11T20:22:48Z

0000-offline-resolution-field.md

+
+  - (Optional) Splitting the resulting `resolved` field into two others: `hash` (the hash of the tarballs) and `tarball` (the tarball name).
+
+  - Adding the hash to the tarball name. This is to prevent issues where two projects are being worked on at the same time, both with the same dependency `foobar`, except that one of those projects is using the development branch of `foobar` (possibly fetching the project directly from git). In such a case, both projects lockfile would resolve to the same dependency name, the same dependency version (because maintainers don't usually bump version number until the release), but different hashes.


arcanis · 2017-03-22T14:12:08Z

I opened a PR for review (yarnpkg/yarn#2970) and updated the rfc (fortunately, there was more to remove than to add 😄).

bestander · 2017-03-22T23:52:01Z

This looks good.
Can you address git dependencies in the RFC?
There we did not just strip source we actually generated a whole new file name some times.

bestander · 2017-03-23T11:46:37Z

Great job, one big source of inconsistent caches is about to go away!

Create 0000-offline-resolution-field.md

0921a34

arcanis changed the title ~~Create 0000-offline-resolution-field.md~~ Prevent Yarn from rewriting the lockfile when switching between online ⇄ offline Mar 7, 2017

bestander reviewed Mar 7, 2017

View reviewed changes

bestander mentioned this pull request Mar 7, 2017

[WIP] Fix git-fetcher offline mirror resolution yarnpkg/yarn#2806

Closed

bestander mentioned this pull request Mar 7, 2017

Some offline-cache dependancies are not being tarballed, and are being inserted into the yarn.lock as remote urls yarnpkg/yarn#2690

Closed

bestander reviewed Mar 11, 2017

View reviewed changes

arcanis mentioned this pull request Mar 22, 2017

[breaking?] - When using offline mirror yarn.lock does not change anymore yarnpkg/yarn#2970

Merged

Update 0000-offline-resolution-field.md

e212657

Move 0000-offline-resolution-field.md

8e20897

bestander merged commit 44f84bb into yarnpkg:master Mar 23, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prevent Yarn from rewriting the lockfile when switching between online ⇄ offline #51

Prevent Yarn from rewriting the lockfile when switching between online ⇄ offline #51

arcanis commented Mar 7, 2017

cpojer commented Mar 7, 2017

bestander Mar 7, 2017

bestander Mar 7, 2017

arcanis Mar 7, 2017

bestander Mar 11, 2017

bestander commented Mar 7, 2017

arcanis commented Mar 7, 2017

bestander commented Mar 7, 2017

arcanis commented Mar 7, 2017

bestander left a comment

bestander Mar 11, 2017

bestander Mar 11, 2017

bestander Mar 11, 2017

bestander Mar 11, 2017

arcanis commented Mar 22, 2017

bestander commented Mar 22, 2017

bestander commented Mar 23, 2017


		Yarn currently has two different type of values for the lockfile `resolved` fields:

		- When online, they're in the form `${source}/${name}-${version}.tar.gz#${hash}`


		- Rational: Why removing the source part of the resolved field?

		The source is redundant, and misleading:


		- If we change our registry URL, the package itself will not have changed. The lockfile should not be modified.

		- The source is already in the package.json file - we can use it when we need to, since we will have checked by then that the resolved version still match (when working with a registry). Worst case scenario (direct link) we will catch that the file will have changed when checking the hash - but that's not something we could have avoided with storing the source in the lockfile anyway.


		- The source is already in the package.json file - we can use it when we need to, since we will have checked by then that the resolved version still match (when working with a registry). Worst case scenario (direct link) we will catch that the file will have changed when checking the hash - but that's not something we could have avoided with storing the source in the lockfile anyway.

		- (Optional) Splitting the resulting `resolved` field into two others: `hash` (the hash of the tarballs) and `tarball` (the tarball name).


		- (Optional) Splitting the resulting `resolved` field into two others: `hash` (the hash of the tarballs) and `tarball` (the tarball name).

		- Adding the hash to the tarball name. This is to prevent issues where two projects are being worked on at the same time, both with the same dependency `foobar`, except that one of those projects is using the development branch of `foobar` (possibly fetching the project directly from git). In such a case, both projects lockfile would resolve to the same dependency name, the same dependency version (because maintainers don't usually bump version number until the release), but different hashes.

Prevent Yarn from rewriting the lockfile when switching between online ⇄ offline #51

Prevent Yarn from rewriting the lockfile when switching between online ⇄ offline #51

Conversation

arcanis commented Mar 7, 2017

cpojer commented Mar 7, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bestander commented Mar 7, 2017

arcanis commented Mar 7, 2017

bestander commented Mar 7, 2017

arcanis commented Mar 7, 2017

bestander left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arcanis commented Mar 22, 2017

bestander commented Mar 22, 2017

bestander commented Mar 23, 2017