lerna publish --canary publishes broken packages

[bimbiltu]

lerna publish --canary only looks at HEAD to determine what packages changed. This means that a package can be published without correct dependencies if those dependencies were changed in an earlier commit. I believe that --canary only operating on the HEAD commit is intentional, but I'm not sure how to achieve the "nightly release" behavior that this flag is supposed to provide.

I read through a few issues about the --canary and this may be a duplicate but I decided to create a new issue to share my use case and describe the problems im running into.

Expected Behavior

Every package published by lerna publish --canary should have dependencies that align with the state of the repo when the publish was run.

Current Behavior

Since lerna publish --canary only looks at the HEAD for changes it can publish broken packages. For an example, see the "Steps to Reproduce" section

Possible Solution

One solution of this could be to add --changed to lerna publish like #962 suggested

As a workaround I could use --force-publish "*" or try to replicate the --since flag using --force-publish and lerna list --since. I could also stop using the --canary flag but then I would lose its ability to automatically increment the -alpha.X suffix.

Steps to Reproduce (for bugs)

Do the following in a monorepo with package-a and package-b where package-a depends on package-b

1. Commit a new feature to package-b and run lerna publish --canary. lerna releases package-a@1.0.0-alpha.0 and package-b@1.0.0-alpha.0. package-a will depend on package-b@1.0.0-alpha.0 as expected.
2. Commit a new feature to package-a and run lerna publish --canary. lerna releases package-a@1.0.0-alpha.1 that depends on package-b@1.0.0. package-a will be broken if it uses a feature introduced in a previous commit and therefore is only available in package-b's canary

lerna.json

I was using a pretty basic config:

{
  "packages": [
    "packages/*"
  ],
  "npmClient": "npm",
  "version": "1.0.1"
}

Context

I have a monorepo containing a few apps and libraries that are used by the apps. I want to run lerna publish --canary in CI for each commit to the master branch and a lot of the apps end up getting broken canaries as they are not using the correct versions of the libraries.

Your Environment

Executable	Version
lerna --version	3.13.4
npm --version	6.4.1
yarn --version	1.13.0
node --version	10.15.1

OS	Version
macOS Mojave	10.14.4

[evocateur]

Yeah, that definitely does seem broken. Part of the problem with "nightly" releases, at least as Lerna attempts them, is that CI often uses a shallow clone, which means in many cases the tags from the previous release literally aren't there. There are workarounds, of course.

At a higher level, I think part of the problem with --canary is that it never had high-enough test coverage to illustrate the intent without ambiguity. I attempted to clear up some of these ambiguities earlier in the 3.x series, but I suspect I just made them "break harder" in use cases like this.

Theoretically, we have "better" coverage now. That being said, I'm positive there are big chunks missing. Awhile back I was hoping that semantic-release could replace lerna publish --canary, but that never really went anywhere once the complexity of lerna versioning was investigated. So I'm pretty open to a lerna release command that could encapsulate this better, but I have a vanishingly small amount of free time available to devote to maintenance (to say nothing of new feature development).

[cspotcode]

I want to achieve essentially the same thing as @bimbiltu. I want changes in any pull request to be published as canary builds to dist-tag PR<pull request ID> so that engineers on my team can propose changes in a PR and immediately start using those changes in their projects.

How can I achieve the desired behavior? Will the following work?

# $PR_NUMBER is set by CI server
gitHash="$(git log -n 1 --format=%h)"
# Bump version numbers using an `PR<pull request ID>-<githash>` preid
lerna version patch --exact --no-push --since master --preid "pr$PR_NUMBER-$gitHash" --yes
lerna publish from-git --dist-tag "pr$PR_NUMBER" --yes

Alternatively, use --since HEAD~1 to get canary behavior of only publishing changes from the most recent commit. I prefer --since master because it avoids situations where I need to commit a useless whitespace change just to trigger publishing a package.

[evocateur]

@cspotcode You can't pass --since to lerna version. As I've mentioned before, in this thread and others, Lerna is not presently optimized for CI automation when publishing or versioning. lerna publish --canary --force-publish --exact --dist-tag "pr${PR_NUMBER}" --preid "pr${PR_NUMBER}" is as close as you're going to get right now.

[erykpiast]

For me, it'd be ideal to have a mechanism for just adding -beta.SOME_UNIQUE_ID to the current version in package.json when publishing. Incrementing the version by patch breaks the flow for me, as the developer is encouraged to increment the version of each package by herself, according to changes she made. But that's a minor issue.

The real one is, at least with private NPM repository based on JFrog Artifactory, that packages with version like x.y.z-beta.n+commit_hash may be published, but cannot be installed with npm i. So I can see (through both npm view and JFrog Web UI) that there is a version available, let's say, 1.2.3-beta.42+abc123, but the NPM client cannot find either 1.2.3-beta.42+abc123 nor 1.2.3-beta.42. The only way I can install such a package is through the dist-tag equal to the name of the git branch that the developer is working on. So, for example, npm i my-package@my_feature_branch passes and resolves URL to the tarball correctly. It's quite a tricky solution, though, as a tag is a moving thing so it always points from the latest version released on the feature branch.

I don't think it's the same problem as @bimbiltu described, but, well, it's about broken packages as well.

[evocateur]

@erykpiast Yeah, in retrospect using the + to append commit_hash was a mistake, as even the public registry completely ignores it (it's allowed to, per the spec).

[cspotcode]

The spec seems to say that anything after the plus does not count towards precedence. However, I would think that if someone types out the full version string, it should find that version exactly. Is the problem that we cannot pass raw version strings to npm, because it always interprets them as a "version range" or "version matcher" or whatever it's called? Does using an equal sign help?

…

On Tue, Dec 31, 2019, 12:55 PM Daniel Stockman ***@***.***> wrote: @erykpiast <https://github.com/erykpiast> Yeah, in retrospect using the + to append commit_hash was a mistake, as even the public registry completely ignores it (it's allowed to, per the spec <https://semver.org/#spec-item-10>). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2060?email_source=notifications&email_token=AAC35OESLYCMX4YA3Y3HZPTQ3OBPVA5CNFSM4HJFUDSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH4P56Q#issuecomment-569966330>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC35OCU75YGVWYPEDGESUTQ3OBPVANCNFSM4HJFUDSA> .

[evocateur]

Yes, true, the spec is only speaking to precedence there. In my manual testing, I found that attempting to publish version 1.2.3-alpha.0+12345 will always result in a version of 1.2.3-alpha.0 being available from the registry. Of course, this was done after i had shipped it... ;_;

The version specifiers passed to npm commands are generally cleaned up by semver's valid() method before further processing, which yields the same behavior as my ad-hoc registry tests.

[dimaShin]

Hi guys.
Thank you for this great tool. It helps me a lot on a daily basis.

I'm facing the same issue and unfortunately didn't get from this thread is there any other than dummy changes in last commit option to publish all changes --since default packages with canary release?

I'm using lerna@3.13.2 but I can use more or less any

[bimbiltu]

@dimaShin The simple workaround is to just pass the --force-publish flag to lerna publish --canary which will make it publish a canary for every package in your monorepo.

If for some reason you don't want to publish all those extra packages you could also write a script that gets the intersection of lerna changed --json (packages changed since last publish) and lerna list --since $GIT_PREVIOUS_SUCCESSFUL_COMMIT^ --include-filtered-dependencies --json (uses jenkins specific var to get packages changed in commits you added to master). This would give you the minimal set of packages to pass to the --force-publish argument.

[NMinhNguyen]

Yes, true, the spec is only speaking to precedence there. In my manual testing, I found that attempting to publish version 1.2.3-alpha.0+12345 will always result in a version of 1.2.3-alpha.0 being available from the registry. Of course, this was done after i had shipped it... ;_;

@evocateur would you accept a PR that removes the +${sha} part or changes it to .${sha} maybe?

lerna/commands/publish/index.js

Line 365 in 3367257

return `${nextVersion}-${preid}.${Math.max(0, refCount - 1)}+${sha}`;

[jakewhelan]

@NMinhNguyen This is sorely needed.

Right now we're using a hack to workaround this, but it's unpleasant:

lerna publish --canary --preid next.$(git rev-parse --short HEAD)

Produces:

my-package@0.0.1-next.604b79a7.4

Where the trailing .4 is the remainder of the (broken) canary versioning.

[NMinhNguyen]

@jakewhelan you may wanna consider patching Lerna just for this bit via https://github.com/ds300/patch-package

[github-actions]

Hi Folks 👋

You may or may not know that lerna is now under the stewardship of Nrwl (announcement here #3121), a company with a long history of not just producing valuable open-source software (OSS), but also backing others (at the time of writing, Nrwl has donated over $50,000 to OSS it hasn't created, see https://opencollective.com/nx for full details).

Quite simply, Nrwl ❤️ OSS, and is committed to making lerna the best it can be. We use it ourselves.

In order to take this awesome project forward from its current state, it is important that we focus our finite resources on what is most important to lerna users in 2022.

With that in mind, we have identified this issue as being potentially stale due to its age and/or lack of recent activity.

---

Next steps:

We want to give you some time to read through this comment and take action per one of the steps outlined below, so for the next 14 days we will not make any further updates to this issue.

@bimbiltu as the original author of this issue, we are looking to you to update us on the latest state of this as it relates to the latest version of lerna.

Please choose one of the steps below, depending on what type of issue this is:

• A) If this issue relates to a potential BUG in the latest version of lerna:

  • Please head over to https://github.com/lerna/repro, fork the repo and apply the changes necessary to reproduce the bug. Then open a PR to that https://github.com/lerna/repro repo and comment back to this thread with the PR number so that we can take a look into the problem and provide a fix.

• B) If this issue is a FEATURE request to be added to the latest version of lerna:

  • Simply comment back on this thread so that we know you still want us to consider the request for the latest version of lerna.

• C) If this issue is a QUESTION which is applicable to latest version of lerna:

  • Please convert the issue to be a Discussion instead: https://github.com/lerna/lerna/discussions

• D) If this issue is no longer applicable to the latest version of lerna:

  • Please close the issue.

---

If we do not hear from @bimbiltu on this thread within the next 14 days, we will automatically close this issue.

If you are another user impacted by this issue but it ends up being closed as part of this process, we still want to hear from you! Please simply head over to our new issue templates and fill out all the requested details on the template which applies to your situation:

https://github.com/lerna/lerna/issues/new/choose

Thank you all for being a part of this awesome community, we could not be more excited to help move things forward from here 🙏 🚀

[bimbiltu]

Great to see lerna is getting a new maintainer!

I've opened a PR with a reproducer for this lerna/repro#2

[afshin-hoseini]

Hey guys,
First of all thanks for the great tool...

I got the same issue here. Any solution?

I was thinking to fetch the latest version according to the provided dist-tag for the other packages except the ones that have been changed ( lerna changed ), right before publishing. 🤔