upload assets on tag #6318

jburel · 2022-04-28T21:10:45Z

Follow up from 5.6.4 release.
This PR adds steps to the build workflow to avoid the manual migration of the artifacts from downloads.o.org to GitHub.

The workflow does the following, When a tag is pushed:

The artifacts are built
A release is created
The artifacts are uploaded

I have opted for one single SHASUMS and MD5 collecting the various values to simplify the selection and reduce the number of uploaded files.

This can replaced the "release" job on CI

See https://github.com/jburel/openmicroscopy/releases/tag/v5.6.4-m1

sbesson · 2022-04-29T14:14:52Z

Immediate thoughts:

using the GitHub release API as the primary source of artifacts feels like a valuable move following the same line of argument as Rtd omero-documentation#2195 (comment),
comparing to https://downloads.openmicroscopy.org/omero/5.6.4/artifacts/, the staging release linked above above omits openmicroscopy-5.6.4.zip. Or is it equivalent the source code zip?
both the build number and the ice separator are removed from the bundle names. I think the former makes sense as it was related to the Jenkins release infrastructure. The latter was meaningful when several Ice versions were concurrently supported so as long as we accept this will not be the case moving forward, it makes sense as a simplification.
With the name simplification, the proposal defines a fairly simple URL for server artifacts https://github.com/ome/openmicroscopy/releases/download/v<version>/OMERO.server-<version>.zip
- is the expectation that all downstream consumers (omero-install, omero-documentation, omego, www.openmicroscopy.org) will be updated to start consuming this new URL? or would we retain some mirroring logic and leave some dependencies on downloads.openmicroscopy.org/omero/?
- this also raises the timeliness of reviewing the repository name as mentioned in Rtd omero-documentation#2195 (comment)

jburel · 2022-04-29T14:24:04Z

openmicroscopy-vXX.zip is equivalent to the source.zip. Note that it is not available on GitHub on the tag 5.6.3/5.6.4
downstream repositories will consume the zip from GitHub (as it was for omero-install prior to 5.6.4). I don't think there is too much to gain to mirror to downloads.o.org
When we have doc on RTD, changing the link and updating the doc will be straightforward but there are not really coupled. Changing to the various repositories could be done ahead of time. This could potentially be done via an action so we keep repositories in synch i.e. new openmicroscopy tag, PR opened in another repo similar to what we do for the doc and omeroweb-install and omero-install

joshmoore · 2022-04-29T15:27:21Z

One thing to keep in mind that I went through with https://github.com/zarr-developers/zarr-logo/pull/2/commits is that on download things get automatically zipped.

Also: I can highly recommend experimenting with the gh release commands to make sure what's being upload and downloaded matches your expectations.

sbesson · 2022-04-29T15:30:01Z

Another component which I forgot to mention in my previous comment is our Ansible deployment stack especially ome.omero_server and the underlying omego library currently rely on the current https://downloads.openmicroscopy.org/omero layout and naming for retrieving the server artifacts. Amongst other, this proposal might force us to revisit ome/omego#127 to download components directly from GitHub.

Something to think about is the breaking impact on deployment if a new library/role release or new scripts are required to upgrade a server. I don't consider it to be a blocker per se but it might lead us to a dedicated upgrade documentation page similar to what we did for the Python3 migration.

jburel · 2022-06-28T15:53:34Z

Reactivate that work

Use a new workflow
Add ice36 to the name
use gh release command
The zip do not get "zipped" twice
See https://github.com/jburel/openmicroscopy/releases/tag/v5.6.5-m1

jburel · 2022-06-28T20:34:05Z

Add openmicroscopy-XXX.zipsee https://github.com/jburel/openmicroscopy/releases/tag/v5.6.5-m2

jburel · 2022-06-29T08:10:25Z

cf. #6318 (comment)
Re-activated see ome/omego#132.

Adding support for latest should prevent the upgrade of link in https://github.com/ome/omero-install/blob/develop/linux/step04_all_omero_install.sh#L15

sbesson

Given the OMERO.server 5.6.5 experience, I am definitely in favor of inverting the release workflow i.e. migrate the artifact generation to GitHub Actions and mirror these artifacts under downloads.openmicroscopy.org.

A few inline comments and a couple of questions:

the new process seems to fully drop the build suffix. Is that on purpose and do we know whether this might cause impact downstream?
we might need to review whether this GH based process might affect a security workflow

.github/workflows/release.yml

jburel · 2022-06-29T19:49:12Z

A few inline comments and a couple of questions:

the new process seems to fully drop the build suffix. Is that on purpose and do we know whether this might cause impact downstream?

I can't get a build number as bXXXX is internal to our CI. The build is now happening in GHA.

sbesson · 2022-06-29T21:13:37Z

I can't get a build number as bXXXX is internal to our CI. The build is now happening in GHA.

I assume in the GitHub context github.run_id would be the closest equivalent. As mentioned above, I do not know of the impact of dropping the build. it An advantage is that would certainly make the download URL fully predictable from the OMERO.server version.

jburel · 2022-06-30T07:23:52Z

Using the job id is an option

jburel · 2022-06-30T08:37:03Z

https://github.com/jburel/openmicroscopy/releases/tag/v5.6.5-m3 using the run_id in the name
This should match the convention currently used when the artifacts are built via Jenkins CI

jburel · 2022-07-01T08:25:22Z

Looking closely, I don't think the build number needs to be kept.
I am also looking into improving ome/omego#132 so we can pass major or major.minor so we can install the latest version from a major version without having to update the link

the usage of get_latest_runs will need to be reviewed. This is mainly (only!) used to retrieve a latest run from Jenkins latest.

jburel · 2022-07-01T12:30:40Z

Looking at another part, candidate for automation, not including the build number will allow to create the url by finding the latest release tag from the GitHub API . Including the build number will not permit that.

joshmoore · 2022-07-04T13:49:47Z

.github/workflows/release.yml

+        run: |
+          set -x
+          assets=()
+          touch SHASUMS


How does the MD5 file that's in your tag get created? Do we need both?

@sbesson and I discussed it and decided to remove the MD5

.github/workflows/release.yml

jburel · 2022-07-05T06:53:42Z

Build number is used in https://github.com/ome/www.openmicroscopy.org/blob/master/omero/downloads/index.html. In that case it already points to GitHub
Ansible role do not rely on build number
Build number in omego is used for downloading from latest CI. The number used in this PR will never match leading to potential confusion i.e. 2 different build numbers

sbesson

One outstanding question about simplifying the for loop.

On the build number suffix, my understanding is that the impact is rather about cleaning up various locations and I am in favor of dropping it.
It partly raises the question of how a rebuild should be handled using GitHub Actions. I assume pushing a new tag?

.github/workflows/release.yml

jburel · 2022-07-06T08:07:25Z

For the build number, it will be matter of adjusting a few places. If we agree, I will roll back and do not add the build number
A rebuild will be happening by pushing a new tag.
One downside of this approach is if we want to retract a release. For that eventually e.g. major releases, rc will be used or possible intermediate dev tags.

jburel · 2022-07-06T13:47:04Z

Last 2 commits

Remove the build number
simplify the code. All actions occur in the target directory so the name in SHASUM does not content target/NAME

Tested via https://github.com/jburel/openmicroscopy/releases/tag/5.6.5-m5

jburel · 2022-07-11T15:41:11Z

Any more comments on this PR?

sbesson · 2022-07-12T07:47:49Z

Is the missing leading 5 in https://github.com/jburel/openmicroscopy/releases/tag/5.6.5-m5 on purpose?

jburel · 2022-07-12T08:19:17Z

oh. Let me check. That should not be the case.

Edit: This is because I did not tag with vX.X.X and I remove the first character i.e. tag_value="${tag_name:1}-ice36"
I will add check if it starts with v to avoid such mistake

jburel · 2022-07-12T08:54:55Z

The last commit has been tested using 2 tags. One starting with v and one without v
https://github.com/jburel/openmicroscopy/releases/tag/5.6.5-m7
https://github.com/jburel/openmicroscopy/releases/tag/v5.6.5-m6

sbesson

Ready to test in the upcoming server release from my perspective

This was referenced Apr 28, 2022

Release workflow #6316

Closed

Rtd ome/omero-documentation#2195

Merged

sbesson mentioned this pull request Jun 22, 2022

remove slice2java #6325

Merged

This was referenced Jun 23, 2022

Build all #6326

Closed

OMERO.server 5.6.5 components #6323

Merged

jburel added 2 commits June 28, 2022 16:18

add release job

bad45e4

run on tag only

0edad46

jburel force-pushed the upload_artifacts branch from 5b5f711 to 0edad46 Compare June 28, 2022 15:51

jburel added 2 commits June 28, 2022 17:16

add release source

7578da2

set properties

18161ff

sbesson reviewed Jun 29, 2022

View reviewed changes

.github/workflows/release.yml Outdated Show resolved Hide resolved

.github/workflows/release.yml Outdated Show resolved Hide resolved

.github/workflows/release.yml Show resolved Hide resolved

jburel added 2 commits June 29, 2022 21:14

remove md5

cc76582

remove conditional

7999012

jburel added 2 commits June 30, 2022 09:09

remove remaining conditional

d9d2035

add run_id to the name

d0b4054

jburel mentioned this pull request Jul 4, 2022

Release process review #6330

Open

joshmoore reviewed Jul 4, 2022

View reviewed changes

.github/workflows/release.yml Outdated Show resolved Hide resolved

sbesson reviewed Jul 6, 2022

View reviewed changes

.github/workflows/release.yml Outdated Show resolved Hide resolved

jburel added 2 commits July 6, 2022 09:19

remove build number

9643483

use wildcard

1969408

add line

bb973e3

jburel mentioned this pull request Jul 11, 2022

use omego ome/omero-install#266

Closed

handle characters before number if any

2945cdf

sbesson approved these changes Jul 25, 2022

View reviewed changes

jburel merged commit 8d20583 into ome:develop Jul 26, 2022

jburel mentioned this pull request Aug 16, 2022

Bump BF version to 6.10.1 ome/omero-model#85

Merged

jburel deleted the upload_artifacts branch March 10, 2023 19:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

upload assets on tag #6318

upload assets on tag #6318

jburel commented Apr 28, 2022 •

edited

Loading

sbesson commented Apr 29, 2022

jburel commented Apr 29, 2022 •

edited

Loading

joshmoore commented Apr 29, 2022

sbesson commented Apr 29, 2022

jburel commented Jun 28, 2022 •

edited

Loading

jburel commented Jun 28, 2022

jburel commented Jun 29, 2022 •

edited

Loading

sbesson left a comment

jburel commented Jun 29, 2022 •

edited

Loading

sbesson commented Jun 29, 2022

jburel commented Jun 30, 2022

jburel commented Jun 30, 2022 •

edited

Loading

jburel commented Jul 1, 2022 •

edited

Loading

jburel commented Jul 1, 2022

joshmoore Jul 4, 2022

jburel Jul 4, 2022

jburel commented Jul 5, 2022 •

edited

Loading

sbesson left a comment

jburel commented Jul 6, 2022 •

edited

Loading

jburel commented Jul 6, 2022

jburel commented Jul 11, 2022

sbesson commented Jul 12, 2022

jburel commented Jul 12, 2022 •

edited

Loading

jburel commented Jul 12, 2022

sbesson left a comment

upload assets on tag #6318

upload assets on tag #6318

Conversation

jburel commented Apr 28, 2022 • edited Loading

sbesson commented Apr 29, 2022

jburel commented Apr 29, 2022 • edited Loading

joshmoore commented Apr 29, 2022

sbesson commented Apr 29, 2022

jburel commented Jun 28, 2022 • edited Loading

jburel commented Jun 28, 2022

jburel commented Jun 29, 2022 • edited Loading

sbesson left a comment

Choose a reason for hiding this comment

jburel commented Jun 29, 2022 • edited Loading

sbesson commented Jun 29, 2022

jburel commented Jun 30, 2022

jburel commented Jun 30, 2022 • edited Loading

jburel commented Jul 1, 2022 • edited Loading

jburel commented Jul 1, 2022

joshmoore Jul 4, 2022

Choose a reason for hiding this comment

jburel Jul 4, 2022

Choose a reason for hiding this comment

jburel commented Jul 5, 2022 • edited Loading

sbesson left a comment

Choose a reason for hiding this comment

jburel commented Jul 6, 2022 • edited Loading

jburel commented Jul 6, 2022

jburel commented Jul 11, 2022

sbesson commented Jul 12, 2022

jburel commented Jul 12, 2022 • edited Loading

jburel commented Jul 12, 2022

sbesson left a comment

Choose a reason for hiding this comment

jburel commented Apr 28, 2022 •

edited

Loading

jburel commented Apr 29, 2022 •

edited

Loading

jburel commented Jun 28, 2022 •

edited

Loading

jburel commented Jun 29, 2022 •

edited

Loading

jburel commented Jun 29, 2022 •

edited

Loading

jburel commented Jun 30, 2022 •

edited

Loading

jburel commented Jul 1, 2022 •

edited

Loading

jburel commented Jul 5, 2022 •

edited

Loading

jburel commented Jul 6, 2022 •

edited

Loading

jburel commented Jul 12, 2022 •

edited

Loading