Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lerna with CI/CD pipeline #1385

Closed
bochen2014 opened this issue Apr 20, 2018 · 8 comments
Closed

lerna with CI/CD pipeline #1385

bochen2014 opened this issue Apr 20, 2018 · 8 comments

Comments

@bochen2014
Copy link
Contributor

@bochen2014 bochen2014 commented Apr 20, 2018

introduction

We have a monorepo project comprised of 5 packages. We are using lerna to manage package versions, with independent strategy. Overall, apart from one particular big surprise, the experience with lerna is good.

our pipeline setup

5 developers work on different features, once their work is finished, code is merged into master, which will trigger a build in Bamboo. The bamboo build server will build the code, bump the version of changed packages and publish them into a private npm registry.

the problem

the problem happens when people merge their code into master while there is an active build in progress.
For example, lets say my project has A@1.0.0, B@1.0.0 and C@1.0.0 to begin with. Here is what happened (in time order)

  1. developer A merged his code into master which contains chore(A) and feat(B). Bamboo picked it up and starts build#1;
  2. while build#1 is under way, developer B merged his code which contains feat(C) and chore(A). This triggers another build#2. Now we have both build#1 and build#2 running at the same time;
  3. build#1 bumped packages to A@1.0.1 and B@1.1.0; and successfully published them into npm registry. But when it tries to git push all those package.json files and changelog.mds back, it failed because it doesn't contains latest change(due to developer B 's merge). So build#1 failed with A@1.0.1 and B@1.1.0 published into npm registry. No commits or tags were pushed to origin.
  4. build#2 bumped packages to C@1.1.0 , A@1.0.1 and B@1.1.0 (note here that B will be picked up as well because build#1 didn't push the new tags. even if it did, build#2 may not be able to see it due to timing condition). It successfully published C@1.1.0 but failed on publishing A@1.0.1 because that package already exists. So build#2 also failed with C@1.1.0 published. No commits or tags were pushed to origin.

Now, imagine you sit in front a dev machine and do a git pull. All your local package.json files are still stating A@1.0.0, B@1.0.0 and C@1.0.0, no new git tags introduced, but all 3 npm packages have been published. What do you do? 😵 😵 😱 😱

possible solutions : disable concurrent builds in bamboo

you may think that by making bambo build sequential the problem may be solved. But it wont' help because build#1 still fails due to not having developer B 's merge. build#2 also will fail because build#1 didn't push its local commits&tags. so disabling concurrent build literally has no effect at all.

your solution

I do have a solution but it's not ideal and I would like to hear from others before I share it here.

I'm pretty sure i'm not alone, as it seems to be the standard use case of lerna. What is the official way of using lerna in a CI/CD pipeline??

@bochen2014 bochen2014 changed the title lerna in production lerna with CI/CD pipeline Apr 20, 2018
@bochen2014
Copy link
Contributor Author

@bochen2014 bochen2014 commented Apr 20, 2018

@evocateur

@bochen2014
Copy link
Contributor Author

@bochen2014 bochen2014 commented Apr 20, 2018

ok. I guess canary release is the way to go.

@bochen2014
Copy link
Contributor Author

@bochen2014 bochen2014 commented Apr 20, 2018

it's hard to understand the purpose of canary flag. in source code , it treat canary same as --skip-git
this.gitEnabled = !(this.options.canary || this.options.skipGit);
so it seems to be a step designed only for npm publish , which is fair enough.

but why do you bump the npm packages version (instead of just suffix with commit short sha1 ), and even default to a minor?

When run with this flag, publish publishes packages in a more granular way (per commit). Before publishing to npm, it creates the new version tag by taking the current version, bumping it to the next minor version, adding the provided meta suffix (defaults to alpha) and appending the current git sha (ex: 1.0.0 becomes 1.1.0-alpha.81e3b443).

how am I supposed to link this npm package back to my git tag? what is purpose of this design?

right. I got it. it basically says "I'm a release candidate for the next minor version" and hence the bump
I'm not still not sure how I can link npm package version with repository version/tag; I suppose I can use npm tag; but I need more investigation

plus, the description is really misleading, as there is no tag created; the entire git is disabled for a canary publish; but the description still says it creates the new version tag by ..... bla bla.
no. there is no new tags created. there is no new commit created. it's purely for npm publish

@evocateur
Copy link
Member

@evocateur evocateur commented Apr 20, 2018

--npm-tag controls the npm package dist-tag, not the annotated git tag (which is indeed skipped with --canary). It's an unfortunately-named option.

--canary was originally intended to be the "nightly" release, run in CI. lerna publish was not originally designed to be run in CI, but it has gained various features over the years. It's certainly the direction that the industry wants to go.

Instead of rewriting lerna publish to be more CI-friendly, I would rather figure out a way to integrate lerna-like versioning decisions into semantic-release, to be honest. It's got a very robust architecture, it's just missing a few of the git calculations and whatnot.

@bochen2014
Copy link
Contributor Author

@bochen2014 bochen2014 commented Apr 21, 2018

thanks @evocateur for your reply. Here is my understanding

  1. semantic-release is not able to manage monorepos and therefore won't be considered
  2. I don't think semantic-release is able to handle concurrent publish either
    the fundamental difference between lerna and semantic-release when speaking to version bump is lerna uses git tag plus packages.json to calculate new version while semantic-release use npm info; its really just a matter of where to read version info from. If we use semantic-release, build#1 will still fail due to not able to push commit back, build#2 is highly likely (not guaranteed though) succeed, but the end result will be missing changelog from build#1.

I think the issue can be easily fixed by putting git push ahead of npm publish in lerna. I created #1387 for it, can you please have a look? thank again.

@bochen2014
Copy link
Contributor Author

@bochen2014 bochen2014 commented Apr 21, 2018

we were quite shocked to know that lerna factor in work tree changes when doing comparison, and that might not be intended.
#1388 solve the issue. please take a look as well

@bdwain
Copy link
Contributor

@bdwain bdwain commented Jul 13, 2018

@bochen2014 we solved this issue by only allowing people to merge to master using a jenkins job, and the jenkins job also auto-rebases the branch onto master before merging it (failing if there is a conflict)

evocateur added a commit that referenced this issue Jul 27, 2018
BREAKING CHANGE:
* `--preid` now defaults to "alpha" during prereleases:

  The previous default for this option was undefined, which led to an awkward "1.0.1-0" result when passed to `semver.inc()`.

  The new default "alpha" yields a much more useful "1.0.1-alpha.0" result. Any previous prerelease ID will be preserved, just as it was before.

* `--no-verify` is no longer passed to `git commit` by default, but controlled by the new `--commit-hooks` option:

  The previous behavior was too overzealous, and the new option operates exactly like the corresponding [npm version](https://docs.npmjs.com/cli/version#commit-hooks) option of the same name.

  As long as your pre-commit hooks are properly scoped to ignore changes in package.json files, this change should not affect you. If that is not the case, you may pass `--no-commit-hooks` to restore the previous behavior.

Fixes #277
Fixes #936
Fixes #956
Fixes #961
Fixes #1056
Fixes #1118
Fixes #1385
Fixes #1483
Fixes #1494
nicolo-ribaudo pushed a commit to babel/lerna that referenced this issue Dec 18, 2018
BREAKING CHANGE:
* `--preid` now defaults to "alpha" during prereleases:

  The previous default for this option was undefined, which led to an awkward "1.0.1-0" result when passed to `semver.inc()`.

  The new default "alpha" yields a much more useful "1.0.1-alpha.0" result. Any previous prerelease ID will be preserved, just as it was before.

* `--no-verify` is no longer passed to `git commit` by default, but controlled by the new `--commit-hooks` option:

  The previous behavior was too overzealous, and the new option operates exactly like the corresponding [npm version](https://docs.npmjs.com/cli/version#commit-hooks) option of the same name.

  As long as your pre-commit hooks are properly scoped to ignore changes in package.json files, this change should not affect you. If that is not the case, you may pass `--no-commit-hooks` to restore the previous behavior.

Fixes lerna#277
Fixes lerna#936
Fixes lerna#956
Fixes lerna#961
Fixes lerna#1056
Fixes lerna#1118
Fixes lerna#1385
Fixes lerna#1483
Fixes lerna#1494
@lock
Copy link

@lock lock bot commented Dec 27, 2018

This thread has been automatically locked because there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked as resolved and limited conversation to collaborators Dec 27, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants