Add support for git-hosted urls as sibling package dependencies #1033

gustaff-weldon · 2017-09-29T09:35:49Z

Description

This PR introduces optional support for package dependency version to be stored as url to git hosted repositories in package.json eg. given package-1 that depends on package-2, a package.json of the former might look like:

// packages/package-1/package.json
{
  version: "0.0.1",
  ...
  dependencies: {
    "package-2": "git://github.com/user/package-2.git#v0.0.1"
  }
}

implementation is based around the idea that package.json file contains a serialized version of package dependency versions, as full urls, that are read internally as semver ones ie. eg. 0.0.1.

uses useGitVersion and gitVersionPrefix options in lerna.json to enable the feature and optionally configure it

{
  "lerna": "2.2.0",
  "packages": [
    "packages/*"
  ],
  "version": "0.0.1",
  "useGitVersion": true,
  "gitVersionPrefix": "v"
}

introduces VersionSerializer that handles deserializing/serializing dependency version from/to package.json and GitVersionParser to handle git-specific urls.

Motivation and Context

Lerna assumes packages to be distributed on npm upon publish. However, when private repo is not an option, packages can be distributed on private GH repositories.

We run a private Github Lerna monorepo with Ember addons and we publish our packages to separate read-only repositories, from which addons are installable separately (and can bring in other subpackages as dependencies).

With versions written down as 0.0.1 installation of git-hosted downstream repositories would fail when resolving other package dependencies. Storing urls in GH forms solves the issues, also keeps history and tags to point to an identical version of package.json files in monorepo and read-only repositories.

This PR shares the work we've done with @synaptiko to streamline this process. We've done our best to make it least invasive. We are open to improvement suggestions to make it even more suitable for inclusion in Lerna.

Known limitations:

feature requires dependency url to contain committish part
publish command should be used with --exact since using ^ or ~ in commitish part is not supported

How Has This Been Tested?

We run code from this PR internally to bootstrap and publish our monorepo. We have also updated Lerna tests to cover added code.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed. (we checked with run ci)

Calanthe · 2017-09-29T11:01:44Z

test/GitVersionParser.js

+    });
+  });
+
+  describe("parseVersion - with prefix", () => {


[minor]: I would rename with prefix into with versionPrefix. The title confused me with the prefix property returned by the parser.

Calanthe · 2017-09-29T11:07:10Z

test/Package.js

+      expect(mockSerializer.serialize.mock.calls.length).toBe(0);
+    });
+
+    it("should call 'deserialize' on old and and 'serialize' on new serializer'", () => {


[minor] double and

Calanthe · 2017-09-29T11:15:08Z

test/PackageGraph.js

+      expect(graph.get(pkg2.name).dependencies).toEqual([pkg1.name]);
+    });
+
+    it(".get should not return the dependencies for urecognized versions", () => {


[minor] unrecognized

Calanthe · 2017-09-29T11:16:51Z

test/PackageGraph.js

+    });
+
+    it(".get should not return the dependencies for urecognized versions", () => {
+      const [pkg1, pkg2] = createPackages("0.0.1", "github:user-foo/project-foo#v0.0.1");


To keep consistency, add spaces before and after [] like in the test before.

Also, I am not sure if you need this test if you have the second .get should not return the dependencies for urecognized versions more detailed test.

Calanthe · 2017-09-29T11:18:58Z

test/PackageGraph.js

+      expect(graph.get(pkg2.name).dependencies).toEqual([]);
+    });
+
+    it(".get should not return the dependencies for urecognized versions", () => {


[minor] unrecognized and spaces before and after []

evocateur

This PR was well-constructed from the get-go, please don't take the number of (mostly minor) comments as implicit criticism! I appreciate the care and attention to detail already in evidence.

Beyond stylistic issues, I think my biggest concern is keeping consistent with our "durable" options pattern (allowing CLI flags to override "embedded" config in lerna.json). As rare as I expect this particular pattern to be, I want to keep these configuration options as consistent as possible.

Thanks!

evocateur · 2017-10-02T18:12:27Z

README.md

@@ -663,6 +663,8 @@ Running `lerna` without arguments will show all commands/options.
    }
  },
  "packages": ["packages/*"]
+  "useGitVersion": true,


These aren't going to be very common, and should be documented separately (perhaps after --use-workspaces). Putting them here just confuses beginners.

evocateur · 2017-10-02T18:13:04Z

README.md

+    name: "my-package-1",
+    version: "1.0.0",
+    bin: "bin.js",
+    dependencies: { "my-package-2": "github:example-user/my-package-2#1.0.0" },


let's spread these dependencies across multiple lines, just like npm formats it.

(the same stuff in various tests is fine on one line)

evocateur · 2017-10-02T18:23:08Z

src/commands/PublishCommand.js

@@ -600,6 +605,7 @@ export default class PublishCommand extends Command {
    });
  }

+


stray newline

evocateur · 2017-10-02T18:35:10Z

src/commands/PublishCommand.js

@@ -156,6 +156,11 @@ export default class PublishCommand extends Command {
    this.gitRemote = this.options.gitRemote || "origin";
    this.gitEnabled = !(this.options.canary || this.options.skipGit);

+    if (this.repository.lernaJson.useGitVersion && !this.options.exact) {


useGitVersion should be referenced from this.options, which supports CLI flag overrides.

That being said, if --use-git-version requires --exact, we should throw an error here.

evocateur · 2017-10-02T18:40:19Z

src/commands/PublishCommand.js

@@ -156,6 +156,11 @@ export default class PublishCommand extends Command {
    this.gitRemote = this.options.gitRemote || "origin";
    this.gitEnabled = !(this.options.canary || this.options.skipGit);

+    if (this.repository.lernaJson.useGitVersion && !this.options.exact) {
+      this.logger.error("config", ```Using git version without 'exact' option is not recommended.


Triple backticks are meaningless in JS, just use a dedent-tagged template literal for a multiline string. (example)

(and really, just copy that example's throwing, don't worry about an explicit this.logger.error call)

evocateur · 2017-10-02T19:04:51Z

src/Package.js

+
+    this._versionSerializer = versionSerializer;
+
+    if (previousSerializer) {


When does this ever happen? Just being defensive?

(It seems strange to assign the versionSerializer multiple times)

Yes, it's just being defensive. Currently, it is not likely that second serializer would be used in one run. YAGNI and remove?

Yes, please.

evocateur · 2017-10-02T19:19:46Z

src/Repository.js

-    this._packageGraph = PackageUtilities.getPackageGraph(this._packages);
+    const packages = PackageUtilities.getPackages(this);
+    const packageGraph = PackageUtilities.getPackageGraph(packages, false, this.lernaJson.useGitVersion
+      ? new GitVersionParser(this.lernaJson.gitVersionPrefix)


I'd prefer the version parser be instantiated outside of the getPackageGraph() arguments, this ternary initially made me think you had forgotten the closing paren. Also, the version parser could be reused in the subsequent version serializer loop.

const { useGitVersion, gitVersionPrefix } = this.lernaJson; // FIXME: should be destructured from command.options to support CLI flag overrides const versionParser = useGitVersion && new GitVersionParser(gitVersionPrefix); const packages = PackageUtilities.getPackages(this); const packageGraph = PackageUtilities.getPackageGraph(packages, false, versionParser); if (useGitVersion) { packages.forEach((pkg) => { pkg.versionSerializer = new VersionSerializer({ monorepoDependencies: packageGraph.get(pkg.name).dependencies, versionParser, }); }); }

evocateur · 2017-10-02T19:20:42Z

src/Repository.js

+    if (this.lernaJson.useGitVersion) {
+      packages.forEach((pkg) => {
+        pkg.versionSerializer = new VersionSerializer({
+          monorepoDependencies: packageGraph.get(pkg.name).dependencies,


graphDependencies sounds clearer to me, here. It's not the dependencies of the entire monorepo, after all.

evocateur · 2017-10-02T19:24:30Z

src/Package.js

  isPrivate() {
    return !!this._package.private;
  }

  toJSON() {
-    return this._package;
+    const pkg = _.cloneDeep(this._package);


Can we move this cloneDeep into the version serializer methods? It seems overly defensive for the non-serialized case.

We can, there's a couple of reasons why this is done like this:

toJSON was returning its internal pkg object, any outside changes on it will affect Package

cloning only for serializer case would make toJSON behave differently - in one case it will return its internal object the other time it would return a deep copy

Despite .toJSON() currently being used only in one place, I think you've raised some good points that will ultimately help future use cases (such as when I actually get around to refactoring lerna into a lerna-managed monorepo and the Package class is exposed in a package of its own). I was being overly-cautious about performance in a place where it really, really doesn't matter (package.json files aren't particularly large, and not deeply nested at all).

Thanks for the clear explanation. :)

evocateur · 2017-10-02T19:38:01Z

src/Repository.js

@@ -118,8 +120,22 @@ export default class Repository {
  }

  buildPackageGraph() {


We should pass an options object derived from this.options when it is called from Command.js and use those properties instead of this.lernaJson. This allows us to support CLI flags (--use-git-version, --git-version-prefix) and eventually argument validation with yargs.

diff --git a/src/Command.js b/src/Command.js index e5bb0c4..032cd01 100644 --- a/src/Command.js +++ b/src/Command.js @@ -288,7 +288,7 @@ export default class Command { } runPreparations() { - const { scope, ignore, registry, since } = this.options; + const { scope, ignore, registry, since, useGitVersion, gitVersionPrefix } = this.options; if (scope) { log.info("scope", scope); @@ -303,7 +303,7 @@ export default class Command { } try { - this.repository.buildPackageGraph(); + this.repository.buildPackageGraph({ useGitVersion, gitVersionPrefix }); this.packages = this.repository.packages; this.packageGraph = this.repository.packageGraph; this.filteredPackages = PackageUtilities.filterPackages(this.packages, { scope, ignore });

(Sorry for the scattered review)

Any suggestion what would be the best way to deal with buildPackageGraph being called from
packages and packageGraph getters?

Could we remove those implicit calls? I do not see any functionality depending on that getter behaviour atm.
We could also set current command options on the repository and then pass them:

// Command.js this.repository.setCurrentCommandOptions(this.options);

But I have mixed feelings about that. Unless there's already some way to access currently running command from the repository?

Yeah, I totally understand the mixed feelings. I have them myself, about the Repository class in general. It's doing too many disparate things for my taste, but it's been low on the priority list to refactor...

So: It occurs to me that Repository itself shouldn't be storing packages and packageGraph in getters since there are literally two places in the entire codebase that they are referenced:

in Command, right after the buildPackageGraph() is called

in UpdatedPackagesCollector, where it violates separation of concern by reaching into (sub-)properties of the command instance just to retrieve mostly-static (i.e., unchanging) values.

Here's what we can do: Move the creation of both the packages list and the packageGraph into Command, where it belongs, and completely remove the get packages and get packageGraph from Repository.

This means all the additional logic added here will be contained inside the place where this.options already exists. Something like this:

diff --git a/src/Command.js b/src/Command.js index 58659bb..aa4bc7f 100644 --- a/src/Command.js +++ b/src/Command.js @@ -294,7 +294,8 @@ export default class Command { } runPreparations() { - const { scope, ignore, registry, since } = this.options; + const { rootPath, packageConfigs } = this.repository; + const { scope, ignore, registry, since, useGitVersion, gitVersionPrefix } = this.options; if (scope) { log.info("scope", scope); @@ -309,10 +310,22 @@ export default class Command { } try { - this.repository.buildPackageGraph(); - this.packages = this.repository.packages; - this.packageGraph = this.repository.packageGraph; - this.filteredPackages = PackageUtilities.filterPackages(this.packages, { scope, ignore }); + const versionParser = useGitVersion && new GitVersionParser(gitVersionPrefix); + const packages = PackageUtilities.getPackages({ rootPath, packageConfigs }); + const packageGraph = PackageUtilities.getPackageGraph(packages, false, versionParser); + + if (useGitVersion) { + packages.forEach((pkg) => { + pkg.versionSerializer = new VersionSerializer({ + graphDependencies: packageGraph.get(pkg.name).dependencies, + versionParser, + }); + }); + } + + this.packages = packages; + this.packageGraph = packageGraph; + this.filteredPackages = PackageUtilities.filterPackages(packages, { scope, ignore }); // The UpdatedPackagesCollector requires that filteredPackages be present prior to checking for // updates. That's okay because it further filters based on what's already been filtered. diff --git a/src/Repository.js b/src/Repository.js index 8316083..7f1e138 100644 --- a/src/Repository.js +++ b/src/Repository.js @@ -8,7 +8,6 @@ import semver from "semver"; import dependencyIsSatisfied from "./utils/dependencyIsSatisfied"; import Package from "./Package"; -import PackageUtilities from "./PackageUtilities"; const DEFAULT_PACKAGE_GLOB = "packages/*"; @@ -67,22 +66,6 @@ export default class Repository { .map(parentDir => path.resolve(this.rootPath, parentDir)); } - get packages() { - if (!this._packages) { - this.buildPackageGraph(); - } - - return this._packages; - } - - get packageGraph() { - if (!this._packageGraph) { - this.buildPackageGraph(); - } - - return this._packageGraph; - } - get packageJson() { if (!this._packageJson) { try { @@ -117,11 +100,6 @@ export default class Repository { return this.version === "independent"; } - buildPackageGraph() { - this._packages = PackageUtilities.getPackages(this); - this._packageGraph = PackageUtilities.getPackageGraph(this._packages); - } - hasDependencyInstalled(depName, version) { log.silly("hasDependencyInstalled", "ROOT", depName, version); diff --git a/src/UpdatedPackagesCollector.js b/src/UpdatedPackagesCollector.js index 67d8ba5..5e51b60 100644 --- a/src/UpdatedPackagesCollector.js +++ b/src/UpdatedPackagesCollector.js @@ -31,7 +31,7 @@ export default class UpdatedPackagesCollector { this.logger = command.logger; this.repository = command.repository; this.packages = command.filteredPackages; - this.packageGraph = command.repository.packageGraph; + this.packageGraph = command.packageGraph; this.options = command.options; } diff --git a/test/Repository.js b/test/Repository.js index 1c21bfc..8b01475 100644 --- a/test/Repository.js +++ b/test/Repository.js @@ -142,32 +142,6 @@ describe("Repository", () => { }); }); - describe("get .packages", () => { - it("returns the list of packages", () => { - const repo = new Repository(testDir); - expect(repo.packages).toEqual([]); - }); - - it("caches the initial value", () => { - const repo = new Repository(testDir); - expect(repo.packages).toBe(repo.packages); - }); - }); - - describe("get .packageGraph", () => { - it("returns the graph of packages", () => { - const repo = new Repository(testDir); - expect(repo.packageGraph).toBeDefined(); - expect(repo.packageGraph).toHaveProperty("nodes", []); - expect(repo.packageGraph).toHaveProperty("nodesByName", {}); - }); - - it("caches the initial value", () => { - const repo = new Repository(testDir); - expect(repo.packageGraph).toBe(repo.packageGraph); - }); - }); - describe("get .packageJson", () => { afterEach(() => { readPkg.sync = readPkgSync;

gustaff-weldon · 2017-10-03T11:08:31Z

@evocateur thanks for a detailed review, much appreciated! I will get back to you once the changes are done.

evocateur · 2017-10-04T16:47:19Z

Golly, I really need to fix that AppVeyor node v4 build. :/

Looking great so far!

…synaptiko

…synaptiko)

gustaff-weldon · 2017-10-05T14:43:09Z

I have updated the code. Please let me know if something can be further improved. If accepted, I will rebase this PR into one commit.

evocateur · 2017-10-05T16:41:45Z

@gustaff-weldon Superb, thank you! Don't worry about rebasing, I will squash merge.

mattbrunetti · 2017-10-05T16:57:10Z

Awesome @gustaff-weldon! I can't wait to try this out!

gustaff-weldon · 2017-10-06T06:38:00Z

@mattbrunetti kudos also to @synaptiko, we worked on that together.
I'm glad you find this useful!

ramasilveyra · 2017-10-06T16:27:10Z

This is awesome! Thanks @gustaff-weldon and @synaptiko!

jhchill666 · 2017-10-07T07:01:53Z

Thanks for the heads up @ramasilveyra

This is fantastic news indeed!

mattbrunetti · 2017-10-11T16:11:18Z

Did not end up using this :( Could not find resources/support on the workflow of this "git-only" way of distributing packages.. Decided to try out npm private packages instead..

synaptiko · 2017-10-11T17:54:32Z

For the future reference, there are multiple ways how to achieve "git-only" distribution from monorepo.

We've tried both of those:

splitsh
regular git filter-branch --subdirectory-filter packages/<package>

Both are quite fast, splitsh is faster (we have less then 200 commits though) and both require to have prepared "downstream" repositories in advance. There is one bummer with filter-branch related to tags. You have to either restore original tags after the push of the "split". Or you have to create a fresh clone of the repository before each split.

@mattbrunetti It was easier for us to setup this than to manage private npm repository (which requires every developer to setup his npm/yarn accordingly).

chinchang · 2018-01-09T05:27:54Z

@synaptiko I have one major confusion regarding all this. Please help me clear it:

"This PR only allows putting versions as git urls. That's all. Pushing individual packages to downstream git repos has to be still handled on our own (using splitish etc)."

Am I correct on this?

synaptiko · 2018-01-09T08:56:05Z

@chinchang yes, you are correct… I understand why it's confusing but unfortunately it would make lerna even more complex to support it directly… it also depends on what CI you use and so on

refs lerna#1033

We now simply recognize any local package name with a matching git url. refs lerna#1033

* Support gitRange (#semver:^1.2.3) as well as gitCommittish * Move dependency update logic into Package#updateDependency() * Test PublishCommand/normal-exact with meaningful fixtures BREAKING CHANGE: * Remove --use-git-version and --git-version-prefix options The former is now implied, and the latter is always "" or "v". Updating git hosted versions no longer throws when --exact is missing. * Remove VersionSerializer, GitVersionParser We now simply recognize any local package name with a matching git url. refs #1033

JacopKane · 2018-08-04T19:10:01Z

As a side tip, I'm experimenting with this tool with git hooks/husky:
https://github.com/ingydotnet/git-subrepo

Seems like doing the job well without the need to keep repositories read-only. The only gotcha was I needed to filter my git hooks for the only root origin in order to get rid of endless push/pulls

lock · 2018-12-27T03:25:22Z

This thread has been automatically locked because there has not been any recent activity after it was closed. Please open a new issue for related bugs.

gustaff-weldon force-pushed the git-version-support branch from 0b64c84 to a97192b Compare September 29, 2017 09:43

Calanthe reviewed Sep 29, 2017

View reviewed changes

gustaff-weldon force-pushed the git-version-support branch 2 times, most recently from 11b4445 to aa85501 Compare September 29, 2017 12:10

evocateur added the Tag: Enhancement label Oct 2, 2017

evocateur suggested changes Oct 2, 2017

View reviewed changes

gustaff-weldon force-pushed the git-version-support branch 2 times, most recently from e96918f to e88357c Compare October 4, 2017 16:00

gustaff-weldon added 4 commits October 5, 2017 15:26

Introduced support for package git dependency urls (coauthored with @…

1e98de4

…synaptiko)

Review fixes

b98e06a

Package graph resolution moved to command; Package cleanup

03dca0d

Readme updates

1aad956

gustaff-weldon force-pushed the git-version-support branch from e88357c to 1aad956 Compare October 5, 2017 13:27

evocateur approved these changes Oct 5, 2017

View reviewed changes

evocateur changed the title ~~Introduced support for git-hosted dependency urls of packages~~ Add support for git-hosted urls as sibling package dependencies Oct 5, 2017

evocateur merged commit 5b8795c into lerna:master Oct 5, 2017

dependencies bot mentioned this pull request Oct 6, 2017

Update lerna from 2.3.1 to 2.4.0 in / storybookjs/storybook#1985

Merged

ramasilveyra mentioned this pull request Oct 6, 2017

Lerna packages installed as private git urls #1003

Closed

dependencies bot mentioned this pull request Nov 17, 2017

Update lerna in / from 2.2.0 to 2.5.1 src-zone/material#17

Closed

zenflow mentioned this pull request Jan 5, 2018

[Question] Using lerna with --use-git-version flag #1179

Closed

renovate bot mentioned this pull request Feb 2, 2018

chore(deps): update dependency lerna to v2.9.0 TylorS/typed-unmaintained#47

Closed

evocateur added a commit to evocateur/lerna that referenced this pull request Feb 20, 2018

Nothing like _actually_ testing code to find the bugs

603d021

refs lerna#1033

evocateur added a commit to evocateur/lerna that referenced this pull request Feb 20, 2018

Remove VersionSerializer

0f15b0d

We now simply recognize any local package name with a matching git url. refs lerna#1033

This was referenced Feb 20, 2018

Add universal hosted git URL support #1278

Merged

Is lerna expected to update versions for git url dependencies as well? #1277

Closed

minhuaF mentioned this pull request Mar 20, 2018

lerna 中文文档 minhuaF/blog#2

Open

kjs3 mentioned this pull request Jun 24, 2018

Monorepo with git-hosted package. yarnpkg/yarn#6027

Open

lock bot locked as resolved and limited conversation to collaborators Dec 27, 2018

		@@ -600,6 +605,7 @@ export default class PublishCommand extends Command {
		});
		}


		this._versionSerializer = versionSerializer;

		if (previousSerializer) {

		@@ -118,8 +120,22 @@ export default class Repository {
		}

		buildPackageGraph() {

Add support for git-hosted urls as sibling package dependencies #1033

Add support for git-hosted urls as sibling package dependencies #1033

Conversation

gustaff-weldon commented Sep 29, 2017 • edited

Description

Motivation and Context

Known limitations:

How Has This Been Tested?

Types of changes

Checklist:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Calanthe Sep 29, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evocateur left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gustaff-weldon Oct 4, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gustaff-weldon commented Oct 3, 2017

evocateur commented Oct 4, 2017

gustaff-weldon commented Oct 5, 2017

evocateur commented Oct 5, 2017

mattbrunetti commented Oct 5, 2017

gustaff-weldon commented Oct 6, 2017

ramasilveyra commented Oct 6, 2017

jhchill666 commented Oct 7, 2017

mattbrunetti commented Oct 11, 2017

synaptiko commented Oct 11, 2017 • edited

chinchang commented Jan 9, 2018 • edited

synaptiko commented Jan 9, 2018

JacopKane commented Aug 4, 2018

lock bot commented Dec 27, 2018

gustaff-weldon commented Sep 29, 2017 •

edited

Calanthe Sep 29, 2017 •

edited

gustaff-weldon Oct 4, 2017 •

edited

synaptiko commented Oct 11, 2017 •

edited

chinchang commented Jan 9, 2018 •

edited