lerna with CI/CD pipeline

[bchenSyd]

introduction

We have a monorepo project comprised of 5 packages. We are using lerna to manage package versions, with independent strategy. Overall, apart from one particular big surprise, the experience with lerna is good.

our pipeline setup

5 developers work on different features, once their work is finished, code is merged into master, which will trigger a build in Bamboo. The bamboo build server will build the code, bump the version of changed packages and publish them into a private npm registry.

the problem

the problem happens when people merge their code into master while there is an active build in progress.
For example, lets say my project has A@1.0.0, B@1.0.0 and C@1.0.0 to begin with. Here is what happened (in time order)

1. developer A merged his code into master which contains chore(A) and feat(B). Bamboo picked it up and starts build#1;
2. while build#1 is under way, developer B merged his code which contains feat(C) and chore(A). This triggers another build#2. Now we have both build#1 and build#2 running at the same time;
3. build#1 bumped packages to A@1.0.1 and B@1.1.0; and successfully published them into npm registry. But when it tries to git push all those package.json files and changelog.mds back, it failed because it doesn't contains latest change(due to developer B 's merge). So build#1 failed with A@1.0.1 and B@1.1.0 published into npm registry. No commits or tags were pushed to origin.
4. build#2 bumped packages to C@1.1.0 , A@1.0.1 and B@1.1.0 (note here that B will be picked up as well because build#1 didn't push the new tags. even if it did, build#2 may not be able to see it due to timing condition). It successfully published C@1.1.0 but failed on publishing A@1.0.1 because that package already exists. So build#2 also failed with C@1.1.0 published. No commits or tags were pushed to origin.

Now, imagine you sit in front a dev machine and do a git pull. All your local package.json files are still stating A@1.0.0, B@1.0.0 and C@1.0.0, no new git tags introduced, but all 3 npm packages have been published. What do you do? 😵 😵 😱 😱

possible solutions : disable concurrent builds in bamboo

you may think that by making bambo build sequential the problem may be solved. But it wont' help because build#1 still fails due to not having developer B 's merge. build#2 also will fail because build#1 didn't push its local commits&tags. so disabling concurrent build literally has no effect at all.

your solution

I do have a solution but it's not ideal and I would like to hear from others before I share it here.

I'm pretty sure i'm not alone, as it seems to be the standard use case of lerna. What is the official way of using lerna in a CI/CD pipeline??

[bchenSyd]

@evocateur

[bchenSyd]

ok. I guess canary release is the way to go.

[bchenSyd]

it's hard to understand the purpose of canary flag. in source code , it treat canary same as --skip-git
this.gitEnabled = !(this.options.canary || this.options.skipGit);
so it seems to be a step designed only for npm publish , which is fair enough.

but why do you bump the npm packages version (instead of just suffix with commit short sha1 ), and even default to a minor?

When run with this flag, publish publishes packages in a more granular way (per commit). Before publishing to npm, it creates the new version tag by taking the current version, bumping it to the next minor version, adding the provided meta suffix (defaults to alpha) and appending the current git sha (ex: 1.0.0 becomes 1.1.0-alpha.81e3b443).

how am I supposed to link this npm package back to my git tag? what is purpose of this design?

right. I got it. it basically says "I'm a release candidate for the next minor version" and hence the bump
I'm not still not sure how I can link npm package version with repository version/tag; I suppose I can use npm tag; but I need more investigation

plus, the description is really misleading, as there is no tag created; the entire git is disabled for a canary publish; but the description still says it creates the new version tag by ..... bla bla.
no. there is no new tags created. there is no new commit created. it's purely for npm publish

[evocateur]

--npm-tag controls the npm package dist-tag, not the annotated git tag (which is indeed skipped with --canary). It's an unfortunately-named option.

--canary was originally intended to be the "nightly" release, run in CI. lerna publish was not originally designed to be run in CI, but it has gained various features over the years. It's certainly the direction that the industry wants to go.

Instead of rewriting lerna publish to be more CI-friendly, I would rather figure out a way to integrate lerna-like versioning decisions into semantic-release, to be honest. It's got a very robust architecture, it's just missing a few of the git calculations and whatnot.

[bchenSyd]

thanks @evocateur for your reply. Here is my understanding

1. semantic-release is not able to manage monorepos and therefore won't be considered
2. I don't think semantic-release is able to handle concurrent publish either
   the fundamental difference between lerna and semantic-release when speaking to version bump is lerna uses git tag plus packages.json to calculate new version while semantic-release use npm info; its really just a matter of where to read version info from. If we use semantic-release, build#1 will still fail due to not able to push commit back, build#2 is highly likely (not guaranteed though) succeed, but the end result will be missing changelog from build#1.

I think the issue can be easily fixed by putting git push ahead of npm publish in lerna. I created #1387 for it, can you please have a look? thank again.

[bchenSyd]

we were quite shocked to know that lerna factor in work tree changes when doing comparison, and that might not be intended.
#1388 solve the issue. please take a look as well

[bdwain]

@bochen2014 we solved this issue by only allowing people to merge to master using a jenkins job, and the jenkins job also auto-rebases the branch onto master before merging it (failing if there is a conflict)

[lock]

This thread has been automatically locked because there has not been any recent activity after it was closed. Please open a new issue for related bugs.