Publish packages in topological order #11434

jenn-le · 2022-08-04T23:45:49Z

This uses lerna ls --toposort in order to get the topological order of packages in the repo and changes the pipeline step to publish packages in this order.

Successful pipeline run here: https://dev.azure.com/fluidframework/internal/_build/results?buildId=83788&view=results

Publish packages in topological order

…-publish

…lish

This reverts commit b45709d.

jenn-le · 2022-08-04T23:49:03Z

Questions:

I noticed we do something different for the pack step when lerna isn't available. Is it necessary to publish in topological order for this scenario as well?
I save the topological order to a file from the pack step so that I don't have to mess with the publish step's working directory in order to get access to lerna. Does this file need to be removed at the end?

tylerbutler · 2022-08-08T21:19:29Z

I noticed we do something different for the pack step when lerna isn't available. Is it necessary to publish in topological order for this scenario as well?

Sort of? If we're not using lerna, then it's an independent package (examples include protocol-definitions, tinylicious, etc.), and thus has its own CI pipeline to build and release. The release process itself should ensure that the releases are done in the right order, (for example, when releasing the client packages, if a protocol-defs update is needed, the release tools require it to be released first) so you shouldn't need to account for it.

I save the topological order to a file from the pack step so that I don't have to mess with the publish step's working directory in order to get access to lerna. Does this file need to be removed at the end?

It will likely trigger the "Extraneous files check" in CI. That's designed to catch files that are generated unexpectedly during CI.

curtisman · 2022-08-15T21:05:17Z

tools/pipelines/templates/include-publish-npm-package-steps.yml

-                package_name=$(grep -oP 'npm notice package: \K.*' "publish_log")
-                npm pack $package_name
-                if cmp -s "$f" "$local_f"
+          f=$(find -name "${packageName}-[0-9]*.tgz")


Does this work with our internal version schema 2.0.0-internal.x.y.z?

Yes, since this is a glob pattern and not regex. This pattern expects a single number after the package name to get around some packages starting with the same names e.g. a-b-2.0.0.tgz and a-b-c-2.0.0.tgz. Then as long as the name ends in .tgz, the file will match.

jenn-le · 2022-08-15T23:58:50Z

I noticed we do something different for the pack step when lerna isn't available. Is it necessary to publish in topological order for this scenario as well?

Sort of? If we're not using lerna, then it's an independent package (examples include protocol-definitions, tinylicious, etc.), and thus has its own CI pipeline to build and release. The release process itself should ensure that the releases are done in the right order, (for example, when releasing the client packages, if a protocol-defs update is needed, the release tools require it to be released first) so you shouldn't need to account for it.

I save the topological order to a file from the pack step so that I don't have to mess with the publish step's working directory in order to get access to lerna. Does this file need to be removed at the end?

It will likely trigger the "Extraneous files check" in CI. That's designed to catch files that are generated unexpectedly during CI.

Thanks, @tylerbutler. Sorry for the late response, I wasn't notified you replied. For the second point, does this check get triggered in test pipelines? The pipelines I've run against this branch have been passing without seeming to trigger anything.

tylerbutler · 2022-08-24T19:09:38Z

For the second point, does this check get triggered in test pipelines? The pipelines I've run against this branch have been passing without seeming to trigger anything.

@jenn-le If the CI is passing, then we'll call it good. :) I suggest merging main and pushing before you merge so you can get a fresh CI run.

tools/pipelines/templates/build-npm-package.yml

…lish

jenn-le · 2022-08-24T22:08:10Z

For the second point, does this check get triggered in test pipelines? The pipelines I've run against this branch have been passing without seeming to trigger anything.

@jenn-le If the CI is passing, then we'll call it good. :) I suggest merging main and pushing before you merge so you can get a fresh CI run.

Here's a fresh, passing CI run: https://dev.azure.com/fluidframework/internal/_build/results?buildId=87938&view=results
I'll go ahead and merge this PR now.

github-actions · 2022-08-24T22:09:45Z

This commit is queued for merging with the next branch! Please ignore this PR for now. Contact @microsoft/fluid-cr-infra for help.

jenn-le and others added 26 commits July 27, 2022 22:49

Publish packages in topological order

2c11ce2

Merge pull request #11319 from jenn-le/topological-package-publish

e09ac7d

Publish packages in topological order

Remove npx

5d78ee1

Test pipeline changes

4f4501e

Merge remote-tracking branch 'upstream/main' into topological-package…

6df85a7

…-publish

Fix working directory of lerna ls call

9739a7c

Merge main

04f5982

Merge branch 'test/topo-package-publish' into topological-package-pub…

037690f

…lish

Fix working directory for lerna command

a6ad213

Test push permissions

79bf7fa

Ensure a tarball is found before trying to publish

c45a785

Fix conditional syntax

6166222

Remove trailing newline

a0bf9ad

Fix path

809b5ff

Add logs to figure out error point

b45709d

Revert "Add logs to figure out error point"

cb70c00

This reverts commit b45709d.

Print package name and file path

c288c40

Check for empty string instead of null

c85e772

Remove files in the correct order

b98c620

Fix syntax

06b42b2

Fix

d4f73f2

Add job

f8091cc

Depend on build stage

f218816

Pass in build

9ccf22b

Remove use of workingDirectory

c31c03b

Remove clean up stage

df860a4

jenn-le requested a review from a team as a code owner August 4, 2022 23:45

github-actions bot added area: build Build related issues base: main PRs targeted against main branch labels Aug 4, 2022

Josmithr requested a review from tylerbutler August 4, 2022 23:56

curtisman reviewed Aug 15, 2022

View reviewed changes

tylerbutler reviewed Aug 24, 2022

View reviewed changes

tools/pipelines/templates/build-npm-package.yml Show resolved Hide resolved

tylerbutler approved these changes Aug 24, 2022

View reviewed changes

jenn-le added 2 commits August 24, 2022 20:58

Add comment

2063fc4

Merge remote-tracking branch 'origin/main' into test/topo-package-pub…

553fde4

…lish

jenn-le merged commit 1c82884 into main Aug 24, 2022

jenn-le deleted the test/topo-package-publish branch August 24, 2022 22:08

tyler-cai-microsoft pushed a commit to tyler-cai-microsoft/FluidFramework that referenced this pull request Aug 26, 2022

Publish packages in topological order (microsoft#11434)

b71f852

WayneFerrao pushed a commit to WayneFerrao/FluidFramework that referenced this pull request Aug 31, 2022

Publish packages in topological order (microsoft#11434)

b165838

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Publish packages in topological order #11434

Publish packages in topological order #11434

jenn-le commented Aug 4, 2022

jenn-le commented Aug 4, 2022

tylerbutler commented Aug 8, 2022

curtisman Aug 15, 2022 •

edited

jenn-le Aug 16, 2022

jenn-le commented Aug 15, 2022

tylerbutler commented Aug 24, 2022

jenn-le commented Aug 24, 2022

github-actions bot commented Aug 24, 2022

Publish packages in topological order #11434

Publish packages in topological order #11434

Conversation

jenn-le commented Aug 4, 2022

jenn-le commented Aug 4, 2022

tylerbutler commented Aug 8, 2022

curtisman Aug 15, 2022 • edited

Choose a reason for hiding this comment

jenn-le Aug 16, 2022

Choose a reason for hiding this comment

jenn-le commented Aug 15, 2022

tylerbutler commented Aug 24, 2022

jenn-le commented Aug 24, 2022

github-actions bot commented Aug 24, 2022

curtisman Aug 15, 2022 •

edited