[rush] publish --suffix does not work as expected

[GeoffreyEmerson]

Summary

The rush publish flag --suffix <suffix> does not work in a way that is useful. It will only add a suffix to the version number in the package.json file when a change file exists, but it does not publish that new version to the npm registry. Subsequent calls to rush version or rush publish will then remove the suffix. It therefor cannot be used as part of a concise publishing strategy.

Repro steps

1. Starting with a change file ready to be processed.
2. Call node common/scripts/install-run-rush.js publish --publish --apply --suffix mySuffix --tag myTag --target-branch main

Expected result:

• "version" in package.json should be updated with new semver based on change file.
• CHANGELOG.md and CHANGELOG.json should be updated with info from change file.
• Change file should be deleted.
• All changes should be added to a git commit and pushed to the target-branch.
• "version" in package.json should be appended with specified suffix.
• Package with suffix should be published to npm registry.
• Package with suffix should not be committed and pushed to the target-branch.

Actual result:

• ❌ "version" in package.json is not updated with new semver based on change file.
• ❌ CHANGELOG.md and CHANGELOG.json are not updated.
• ❌ Change file is not deleted.
• ❌ Expected changes are not added to a git commit and pushed to the target-branch.
• 👍 "version" in package.json is appended with specified suffix.
• ❌ Package with suffix is not published to npm registry.
• ❌ Package with suffix is improperly committed and pushed to the target-branch.

Details

If there is a use-case for adding a suffix to "version" in package.json and nothing else, I can't figure out what it is.

rush version --bump or rush publish do not play well with an existing suffix in the "version" field of the package.json. They appear to consider removing the suffix to be a patch update, so the semver doesn't change. Therefor it seems to me that the suffixed version of the published package version should simply not be committed.

If the use-case is not to update the semver and delete the change file, the flags --partial-prerelease --prerelease-name <suffix> seem to work acceptably well for that. However this is not a viable workaround for the use-case of publishing a new version with a suffix, because a rush version --bump deletes the change file, which causes the publish --partial-prerelease to fail.

Minimal repo for testing: https://github.com/GeoffreyEmerson/rush_test

Standard questions

Question	Answer
@microsoft/rush globally installed version?	None, I use the common/scripts
rushVersion from rush.json?	5.88.0
useWorkspaces from rush.json?	No
Operating system?	MacOS 12.6
Would you consider contributing a PR?	Not sure what the fix is
Node.js version (node -v)?	16.16.0

[iclanton]

From the (pretty lacking) description of the PR that introduced this feature (#143), it's noted that the suffix arg is used to just include a suffix, but not bump versions. If you're looking to do a prerelease, (with the style 1.2.3-prerelease), it looks like --prerelease-name is what you'd be looking for.

Can you describe your scenario in some more detail? Are you trying to do prereleases (with a leading -), or metadata (with a leading +)? Are you looking to release a hotfix for certain packages? If that's the case, look into the hotfixChangeEnabled flag in rush.json.

As an aside @elliot-nelson has been talking about a new design for publishing. Adding him here in case he has an insight and/or if this issue is relevant to the new design.

[elliot-nelson]

Thanks Ian, I'm very interested in the use case!

If you're following semver strictly (which is not always exactly what you need), then what comes after the dash is a prerelease meaning the "order" is 1.1.1, then 1.1.2-alpha1, then 1.1.2-alpha2, then 1.1.2-alpha3, then 1.1.2 (the "suffixes" come before the canonical release of the same name).

So from this standpoint, when you are publishing a prerelease, you'd usually use an alternate workflow that temporarily consumes changefiles to produce a changelog, but does not actually commit those changes, leaving the change files unconsumed in main.

So you might do:

Command	Version Produced	Permanent (changes pushed to main)
rush publish	1.1.1	Yes
rush publish --prerelease-name alpha1	1.1.2-alpha1	No
rush publish --prerelease-name alpha2	1.1.2-alpha2	No
rush publish	1.1.2	Yes

It's intentional that nothing about "alphas" is checked into the repo, from the standpoint of the official commit history, the changelog went from 1.1.1 -> 1.1.2.

Is your use case significantly different than this? Are you using prerelease/suffix to mean something different?

[GeoffreyEmerson]

Scenario: We have a number of npm packages in a rush monorepo that are in use in various places in our company. In the coming months, we are testing the possibility of making some fairly major changes and upgrades to these packages. So we begin making those changes on an alpha branch. It is likely that one or more consumers of these packages will begin using the alpha versions in some way, so we need to publish them. I'd like to publish them as "v4.0.0-alpha" or similar. And we'd like to maintain our normal CI processes, so that a new alpha release would be something like "v4.0.1-alpha" or "v4.0.0-alpha1".

The development process, it seems to me, should be practically identical to a main trunk release.

1. Developer commits a code change on a feature branch and creates a change file using rush change.
2. Developer pushes code to git repo and creates a PR/MR to alpha branch.
3. PR gets tested, approved, and merged to alpha branch.
4. CI bumps the version using change file(s) and deletes the change file.
5. CI appends the suffix.
6. CI publishes new npm package version.

I'm basically looking for a totally normal rush publish command, except with a suffix added.

So from this standpoint, when you are publishing a prerelease, you'd usually use an alternate workflow that temporarily consumes changefiles to produce a changelog, but does not actually commit those changes, leaving the change files unconsumed in main.

Let me think through how this would work.

1. rush publish --apply without a target-branch would make the changes and leave them uncommitted.
2. Commit the change logs
3. Revert the other changes file with git reset –hard
4. rush publish --publish --apply --tag alpha -partial-prerelease --prerelease-name alpha --target-branch alpha applies the semver and prerelease-name, but doesn't delete the change file.
5. rush version --bump --target-branch alpha which doesn't delete the change file for some reason, but it does trim the suffix from the version.
6. Manually delete the change file(s) and commit

Pretty clunky, but it might do what I want.

[elliot-nelson]

@GeoffreyEmerson Thanks for the details... indeed it is a bit clunky.

I'll offer a couple options, let me know which one you think fits best with what you'd want... I think your scenario can be simplified slightly.

1. Change-tracked publishing, from alternate branch

In this scenario you're looking for a branch -- we'll say alpha -- that acts like a whole separate main branch. People rack up change files in this branch, and every night your publish job runs and consumes them, you just want it all namespaced away from your real publishing.

BRANCH_NAME=alpha

# Use (but don't consume) change files, update version and add prerelease, then publish
rush publish --publish --tag $BRANCH_NAME --prerelease-name $BRANCH_NAME --apply

# Throw away all changes
git reset --hard HEAD

# Do a "normal" bump and commit
rush version --bump -b $BRANCH_NAME --ignore-git-hooks --bypass-policy

In this model your CHANGELOG.md will be a bit misleading because it will contain change entries for version 1.1.0, 1.2.0, 1.3.0, 1.4.0, etc., although the actual versions published will be 1.1.0-alpha, 1.2.0-alpha, 1.3.0-alpha, and so on. That's fine as long as you never intend to be able to merge this branch "as-is" into main (which would probably be a pretty bad idea anyway, for a long-running feature branch like this; I imagine at some point you would dismantle the branch into bite-size PRs back to main so they can get proper reviews).

2. Periodic publishing, no change tracking

If you just need a way to shove latest alphas to people, but nobody is going to be looking at change logs, it's a lot simpler -- you can just use unique prerelease tags.

BRANCH_NAME=alpha
PRERELEASE=$BRANCH_NAME-$(date +%s)

# Use (but don't consume) change files, update version and add prerelease, then publish
rush publish --publish --tag $BRANCH_NAME --prerelease-name $PRERELEASE --apply

# Throw away all changes
git reset --hard HEAD

In this model, you would rack up change files in common/changes forever without consuming them, and wouldn't ever check in change logs -- but you would use those change files to determine what to publish, based on the predicted eventual update. So the published versions for a given package would look something like:

• 1.1.0-alpha-1673501954
• 1.1.0-alpha-1673514913
• 1.1.0-alpha-1673529933
• 2.0.0-alpha-1673539955 (because you racked up at least one "major" change)

Unlike Option 1, this workflow would not be idempotent -- every time you run it, it shoves out a package, even if nobody made changes. That's fine if it's manual, maybe not fine if it's scheduled nightly. However, the advantage is that the diff from alpha -> main represents a "real" collection of change files (although again, I don't know that I'd recommend a huge merge like that).

[GeoffreyEmerson]

I appreciate all the suggestions. After looking at them and experimenting a bit more, here's what I've settled on for now:

Note, this is in GitLab CI.

WORKING_BRANCH is just a temporary branch created for the CI script to work in. I set it as alpha-publish-${CI_COMMIT_SHORT_SHA} so that it's not going to collide or overwrite anything accidentally.

"alpha-branch" is the name of my long-lived experimental branch in the repo.

In script:

# Workaround for publishing with a suffix
- git checkout -b ${WORKING_BRANCH}
# dryrun publish to create changelogs
- node common/scripts/install-run-rush.js publish --apply --add-commit-details
# commit changelogs
- git add '**/CHANGELOG.*'
- git commit -m "preserve changelogs"
# revert other changes
- git checkout -- .
# perform a prerelease style publish; this step is where the npm package(s) are actually published to the registry
- node common/scripts/install-run-rush.js publish --publish --apply --partial-prerelease --prerelease-name alpha --tag alpha --ignore-git-hooks
# revert package.json changes as they are not needed and ruin the "workspace:alpha" usage
- git checkout -- .
# run version bump to get clean updated semver in package.json files
- node common/scripts/install-run-rush.js version --bump --ignore-git-hooks
# delete change files
- rm -rf ./common/changes
- git add .
- git commit -m "save changes"
# push changes to repo
- git checkout alpha-branch
- git merge --squash ${WORKING_BRANCH}
- git commit -m "alpha-branch version bump [skip ci]"
- git branch -D ${WORKING_BRANCH}
- git push origin alpha-branch

Edit: I've discovered this is also not working as planned, because rush version --bump is changing "workspace:alpha" to specific version numbers. I'm probably going to abandon this workflow completely and just allow change files to accumulate, which appears to be the expected workflow for upcoming versions.

[elliot-nelson]

👍 Thanks for the update, interesting to see the final workflow.

I'll note this down for the eventual rewrite, but I don't know if this style of change tracking would become much different; it's bit of an outlier because eventually, presumably, the experiment will be successful and you'll want to merge everything to main -- but from main's perspective, all of the changelog can't be merged into the main changelog unless you throw it away, collapse it into a single entry, or spend time rewriting it to "rebase" the changes into the main changelog. So in a way, this is an odd workflow.

That said, I do think it should be a lot easier to "permanently" suffix everything in a branch, and apply normal rush commands like rush version and rush publish, never removing the -suffix for that branch. Maybe this could be supported by publishing somehow -- it's similar to the hotfix functionality Ian mentioned, but you intend it to be applied to all future versions from that branch and not just one.

[GeoffreyEmerson]

When the time comes to go from alpha to latest, yes I expect to squash alpha-branch. This isn't something I've done before, but since I'm also porting changes to latest into alpha-branch, hopefully that won't be too bad. Another possibility, though it might not be practical, is to simply rename main to legacy (or something like that) and then rename alpha-branch to main.

I agree that it would be great if rush version and rush publish would respect existing suffixes instead of the current behavior which is again not following through with all the expected tasks of those commands.

[GeoffreyEmerson]

@elliot-nelson Thanks again for the advice. After trying a few different things, I decided to adjust my expectations and go with the "Periodic publishing, no change tracking" option you described. It's working well, now that I understand the intended workflow.