MONGOSH-122: submit to barque #286

lrlna · 2020-08-13T13:29:26Z

Description

This PR introduces curator, a tool maintained by the build team to upload bundled and packaged assets to barque, MongoDB's PPA service.

Package passed to curator is from Evergreen's S3 bucket. Curator is only invoked when running on Debian , Ubuntu, and Redhat (rhel) build variants for the moment. BARQUE_USERNAME and BARQUE_API_KEY are currently stored in Evergreen and are extracted the same way as other env vars. Similar strategy is used for NOTARY_TOKEN , which are necessary for signing the package for a specific mongodb version. We are currently only signing for 4.4.0.

The above variables, BARQUE_USERNAME, BARQUE_API_KEY, NOTARY_TOKEN, NOTARY_KEY_NAME are read by curator as env vars (using child_process.execFile env option), so we do not have pass them on as arguments to the curator CLI. Not passing them as arguments also prevents them from being leaked to our task logs.

Open Questions

repo-config.yml is currently only supported for 1 ubuntu, 1 debian, 1 redhat platforms. This should perhaps be expanded to how mongo-tools are handling this.
This is currently only building for amd64 architecture for Debian and Ubuntu, and x86_64 for Redhat. (can also be x86_64 very easily). I am not sure whether I should be reading current build variant's architecture, or specifically setting different architectures based on what is described in repo-config.yml.
We are only building for 4.4.0 mongodb version. I am not sure whether we should expand to other versions. Depending on which ones we have to support, we will have to adjust the NOTARY_TOKEN and NOTARY_KEY_NAME, as those only work for 4.4.0 series. (@mmarcon)
We are only building for org mongodb edition. I am not sure if we should only be building for enterprise, or both! (@mmarcon)
Currently each linux-like Build Variant will make its own barque request (via curator). Meaning, ubuntu will send off ubuntu, debian will send off debian. This is useful, as we are able to right away grab the applicable s3 path for the package, and send that with the request. I am not sure if we should be considering doing them all at once. It'll be a bit more work to get all the correct s3 paths, but if that's what others are doing, we should do the same. From my (very very basic) understanding of golang, it seems like mongo-tools is also doing it build-variant by build-variant. I could be wrong!

packages/build/src/evergreen.ts

config/repo-config.yml

Kimchelly

The repo config LGTM

config/repo-config.yml

mmarcon · 2020-08-27T14:12:17Z

@lrlna A few questions:

Aren't amd64 and x86_64 the same thing? https://en.wikipedia.org/wiki/X86-64
you say "We are only building for 4.4.0 mongodb version": what does that mean exactly? Same question for "We are only building for org mongodb edition".

I feel we should probably have a quick chat about all of this, it's probably easier than discussing it in comments.

mcasimir

Looks great! Please check the execCurator function, looks like we are not awaiting the promise there.

packages/build/src/barque.ts

packages/build/src/build-and-release.ts

scripts/import-expansions.js

rcsanchez97 · 2020-09-16T18:08:54Z

@lrlna The changes look good to me. The only comment I have relates to your open question 5. For libmongocrypt our approach is similar to the approach you took here. The primary difference achieved by aggregating the publication is that the publication to barque only happens if all of the various variants successfully build their packages for a given Evergreen build. If it is more desirable to have as many packages published as possible for any given build, then your current approach is the best. If, however, it is more desirable to only publish when all variants have built their respective packages (and by implication this means that you wait for slow variants before publishing and any failure to build packages on any variant results in no publication), then aggregating makes more sense. One additional consideration is that since mongosh is closely tied to the server, you might consider adopting their approach (if it is different from the current) for consistency.

lrlna · 2020-09-22T15:30:20Z

packages/build/src/barque.ts

+
+    if (this.config.platform === Platform.Linux) {
+      try {
+        await this.execCurator(tarballURL, repoConfig);


try/catching this here so I can test this function easier. Still throw an error in the catch, so we can abort the release process if this is unsuccessful.

packages/build/src/barque.spec.ts

package.json

packages/build/src/barque.spec.ts

packages/build/src/barque.ts

- Introduces `curator` as a tool to upload packaged assets to barque, MongoDB's PPA service. - Uses an already bundled and packaged mongosh binary that was already uploaded to Evergreen's S3 bucket to send to barque. - Adds additional environmental variables to the build process: `BARQUE_USERNAME`, `BARQUE_API_KEY`, `NOTARY_TOKEN`, `NOTARY_KEY_NAME`. These are then used by `curator`. - Adds `repo-config.yml` that is necessary for a barque upload. The config specifies uploads for 3 different distros: `debian10`, `ubuntu1804`, `rhel80`. - We only upload the package to `org` MongoDB repository. - We only upload along with `4.4.0` MongoDB version. - These artifacts are only sent to `barque` when we do a public release (`shouldDoPublicRelease`), i.e. a tagged commit and on main branch. - We fetch the latest curator tarball before proceeding to upload to barque.

addaleax

Nice :)

packages/build/src/build-and-upload.spec.ts

addaleax · 2020-09-25T09:59:19Z

packages/build/src/barque.ts

+  Ubuntu = 'amd64',
+  Debian = 'amd64',
+  Redhat = 'x86_64',
+}


It feels a bit odd to infer the architecture name from the distro name – maybe we should add a buildArch key to the config, and then take that into account here as well (in determineArch())? It’s not important for this PR, but maybe the comment above could mention it?

the config is created based on the variables that evergreen provides for us, so if I were to put arch in the config, I would be basically doing the same thing as I am here. Which perhaps is not actually bad since we get to keep everything in one place?

We could also just make a ticket (or just discuss in design doc) for the release epic to just stop detecting what to do based on which runner we are on and rather pass those stuff as parameters in the scripts / as env var and make that populate the config.

What everyone thinks about that? Is probably not going to change much if we do something only here.

I think that it could be discussed in that next release automation epic!

mcasimir · 2020-09-25T10:01:00Z

packages/build/src/build-and-upload.spec.ts


  [true, false].forEach((isPublicRelease) => {
-    it(`uploads the artifact to evergreen if is ${isPublicRelease ? 'a' : 'not a'} public release`, async() => {
+    it(`uploads the artifact to evergreen if is ${isPublicRelease ? 'a' : 'not a'} public release`, async () => {


I think we need a test to check we only release to barque if isPublicRelease
and one to check that we don't if is not a public release or?

are you saying you want a test that's like this but for does not release to barque if not a public release? Just triple checking!

oh yep, thanks!

Co-authored-by: Anna Henningsen <anna.henningsen@mongodb.com>

lrlna force-pushed the submit-to-barque branch from 282f2d5 to 5b51c19 Compare August 14, 2020 11:42

mcasimir reviewed Aug 14, 2020

View reviewed changes

packages/build/src/evergreen.ts Outdated Show resolved Hide resolved

lrlna force-pushed the submit-to-barque branch from 8bc46b6 to d87bcb0 Compare August 14, 2020 13:45

lrlna changed the title ~~[WIP] submit to barque~~ [WIP] MONGOSH-122: submit to barque Aug 17, 2020

lrlna force-pushed the submit-to-barque branch 5 times, most recently from 8c130a3 to ed7e321 Compare August 26, 2020 11:19

lrlna changed the title ~~[WIP] MONGOSH-122: submit to barque~~ MONGOSH-122: submit to barque Aug 26, 2020

lrlna commented Aug 26, 2020

View reviewed changes

config/repo-config.yml Show resolved Hide resolved

Kimchelly approved these changes Aug 26, 2020

View reviewed changes

config/repo-config.yml Show resolved Hide resolved

mcasimir requested changes Sep 1, 2020

View reviewed changes

packages/build/src/barque.ts Outdated Show resolved Hide resolved

packages/build/src/build-and-release.ts Outdated Show resolved Hide resolved

scripts/import-expansions.js Outdated Show resolved Hide resolved

lrlna force-pushed the submit-to-barque branch from ed7e321 to a91b263 Compare September 21, 2020 19:38

lrlna commented Sep 22, 2020

View reviewed changes

addaleax approved these changes Sep 22, 2020

View reviewed changes

packages/build/src/barque.spec.ts Show resolved Hide resolved

package.json Show resolved Hide resolved

mcasimir reviewed Sep 22, 2020

View reviewed changes

packages/build/src/barque.spec.ts Outdated Show resolved Hide resolved

mcasimir reviewed Sep 22, 2020

View reviewed changes

packages/build/src/barque.ts Outdated Show resolved Hide resolved

lrlna force-pushed the submit-to-barque branch from c1b9698 to d0dff34 Compare September 24, 2020 15:56

export evergreen artifact path to be submitted to barque

d7d880f

lrlna force-pushed the submit-to-barque branch from 05d2cbf to 0b6018f Compare September 25, 2020 09:47

lrlna force-pushed the submit-to-barque branch from 0b6018f to 9bc3d93 Compare September 25, 2020 09:48

addaleax approved these changes Sep 25, 2020

View reviewed changes

mcasimir reviewed Sep 25, 2020

View reviewed changes

Update packages/build/src/build-and-upload.spec.ts

9788925

Co-authored-by: Anna Henningsen <anna.henningsen@mongodb.com>

mcasimir approved these changes Sep 25, 2020

View reviewed changes

additional tests

ddee27e

Merge branch 'master' into submit-to-barque

e3c1ac2

lrlna merged commit 4cf2e7e into master Sep 29, 2020

lrlna deleted the submit-to-barque branch September 29, 2020 10:11

MONGOSH-122: submit to barque #286

MONGOSH-122: submit to barque #286

Uh oh!

Conversation

lrlna commented Aug 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Open Questions

Uh oh!

Uh oh!

Uh oh!

Kimchelly left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mmarcon commented Aug 27, 2020

Uh oh!

mcasimir left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rcsanchez97 commented Sep 16, 2020

Uh oh!

lrlna Sep 22, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

addaleax left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

addaleax Sep 25, 2020

Choose a reason for hiding this comment

Uh oh!

lrlna Sep 25, 2020

Choose a reason for hiding this comment

Uh oh!

mcasimir Sep 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lrlna Sep 25, 2020

Choose a reason for hiding this comment

Uh oh!

mcasimir Sep 25, 2020

Choose a reason for hiding this comment

Uh oh!

lrlna Sep 25, 2020

Choose a reason for hiding this comment

Uh oh!

mcasimir Sep 25, 2020

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

lrlna commented Aug 13, 2020 •

edited

Loading

mcasimir Sep 25, 2020 •

edited

Loading