Publishing artifacts #1541

udosson · 2021-05-03T14:52:05Z

This PR implements the publishing of the artifacts to the JFrog Artifactory of Hyperledger..

The PR currently works for ubuntu 16.04 only.

A successful run of the GitHub action with publishing the artifacts to the JFrog Artifactory of Hyperledger can be found here.

An update of the Docker image was necessary to include zstd which is required to pass a cache created by a container to a job that natively runs on ubuntu.

sovbot · 2021-05-03T14:52:06Z

Can one of the admins verify this patch?

WadeBarnes · 2021-05-10T17:32:42Z

(ci) test this please

WadeBarnes

Looking good.

I think the pool directory structure for the 3rd party libraries should follow the Debian conventions rather than having them all published under the indy/pool/${RELEASE}/i/indy-plenum/ folder.

For example:

python3-base58_1.0.0_amd64.deb => indy/pool/${RELEASE}/b/base58/python3-base58_1.0.0_amd64.deb
python3-jsonpickle_0.9.6_amd64.deb => indy/pool/${RELEASE}/j/jsonpickle/python3-jsonpickle_0.9.6_amd64.deb
python3-rocksdb_0.6.9_amd64.deb => indy/pool/${RELEASE}/r/rocksdb/python3-rocksdb_0.6.9_amd64.deb
rocksdb_5.8.8_amd64.deb => indy/pool/${RELEASE}/r/rocksdb/rocksdb_5.8.8_amd64.deb
python3-ursa_0.1.1_amd64.deb => indy/pool/${RELEASE}/u/ursa/python3-ursa_0.1.1_amd64.deb

WadeBarnes · 2021-05-11T13:45:27Z

.github/workflows/build.yaml

-          mkdir -p /tmp/build-output
-          ./build-scripts/ubuntu-1604/build-indy-plenum.sh /__w/indy-plenum/indy-plenum 1.14.0 /tmp/build-output
+          mkdir -p /tmp/plenum-build
+          ./build-scripts/ubuntu-1604/build-indy-plenum.sh /__w/indy-plenum/indy-plenum 1.14.0 /tmp/plenum-build


Hardcoded version number.

Thanks for the feedback!
In the current repo of sovrin the 3rd party libraries are organized as follows
python3-base58_1.0.0_amd64.de => deb/pool/xenial/master/p/python3-base58/python3-base58_1.0.0_amd64.deb
python3-rocksdb_0.6.9_amd64.deb=> /deb/pool/xenial/master/p/python3-rocksdb/python3-rocksdb_0.6.9_amd64.deb
rocksdb_5.8.8_amd64.deb => deb/pool/xenial/master/r/rocksdb/rocksdb_5.8.8_amd64.deb
---> All python packes in deb/pool/xenial/master/p/python3-*
Shall we replicate the current directory structure are change it to the proposed structure of @WadeBarnes?

And yes, the version was and is hardcoded. Is there any file where we maintain the current version of plenum? Then we could extract the version from there.

@udosson, I had noticed the same thing with the python packages all being published under .../p/python*/. I'm not really a fan of that convention. Taking the rocksdb dependencies for example, I find it much more intuitive to have the rocksdb dependencies grouped together:

indy/pool/${RELEASE}/r/rocksdb/python3-rocksdb_0.6.9_amd64.deb indy/pool/${RELEASE}/r/rocksdb/rocksdb_5.8.8_amd64.deb

Then it's clear there is a dependency on rocksdb as well as the python module for rocksdb.

https://github.com/hyperledger/indy-plenum/blob/master/plenum/__version__.json is the version file maintained by the https://github.com/hyperledger/indy-plenum/blob/master/bump_version.sh script used to update the package version.

WadeBarnes · 2021-06-16T21:36:02Z

@udosson, I'm testing some updates to this process. I've created a script that will upload the packages to the repository in the manner I've described. The same script should work for the plenum and 3rd party artifacts.

WadeBarnes · 2021-06-21T22:19:24Z

A little bit of clean-up to do but I have it working. Successful publishing run here, and the resulting repo structure can be found here.

swcurran · 2021-06-21T22:20:44Z

w00t!!!!! Well done!

WadeBarnes · 2021-06-22T19:23:10Z

I dug a little further into why repeated build/upload of indy-plenum packages fail, yet repeated build/upload of the 3rd party packages succeed. It seems the indy-plenum build produces a different checksum, even when the exact same code is used for the build. That appears to be causing issues with the lookup the jfrog cli does before it uploads.

WadeBarnes · 2021-06-22T20:12:44Z

The only files that change from build to build are PKG-INFO and __manifest__.json. It's not the content that changes, just the order of the content in the file; however this affects the checksum of the file(s) and therefore the checksum of the overall package.

WadeBarnes · 2021-06-22T22:46:47Z

fpm is also including a dummy changelog that has a date and time stamp which is making every package different too.

WadeBarnes · 2021-07-05T19:07:42Z

@udosson, the changes here (WadeBarnes@b6b6378) are ready to go. I'll rebase them onto this PR once you squash your commit history.

WadeBarnes · 2021-07-05T19:57:18Z

I've cleared out all of the dev builds I did for indy-plenum and left the 3rd party dependencies because they won't cause issues. The one package I left behind, https://hyperledger.jfrog.io/ui/repos/tree/General/indy%2Fpool%2Fdev%2Fi%2Findy-plenum%2Findy-plenum_1.13.0.dev.0_amd64.deb, is a 20.04 package Devin did for testing CI/CD processes on the Sovrin side.

Signed-off-by: udosson <r.klemens@yahoo.de>

udosson · 2021-07-06T09:17:42Z

@udosson, the changes here (WadeBarnes@b6b6378) are ready to go. I'll rebase them onto this PR once you squash your commit history.

@WadeBarnes, I squashed my commit history. You're good to rebase your commits onto this PR.

WadeBarnes · 2021-07-06T12:28:54Z

(ci) test this please

WadeBarnes · 2021-07-06T14:52:02Z

(ci) test this please

WadeBarnes · 2021-07-06T15:59:35Z

To those those marked as reviewers; @Toktar , @ianco , @m00sey , @ryjones , @mirgee. We're looking more for a sanity check. Some extra eyes on the code to maybe pick up something we missed or point out a better way of doing something.

ryjones

From a 10,000 foot view - LGTM

m00sey

I haven't looked at this in a while, and I've never used jFrog - at first glance there is nothing alarming. LGTM

WadeBarnes · 2021-07-06T17:20:33Z

I haven't looked at this in a while, and I've never used jFrog - not at first glance there is nothing alarming. LGTM

Great. The jFrog part has been through a LOT of testing.

ianco

Looks good to me

- Implement organized publishing of Debian packages. - Set the build version dynamically based on the module version and the type of build being performed. - Migrates the functionality of the following [sovrin-foundation/jenkins-shared](https://github.com/sovrin-foundation/jenkins-shared.git) libraries into a GitHub action. - `getReleaseVersion.groovy` - The `Set release versions` stage from `testAndPublish.groovy` - Cache pip packages to speed up installing the dependencies on the test containers. - This is a bit of a compromise since dependencies could change from build to build making image caching complicated. - Maintain a sorted order for the manifest. - Update `build` and `lint` image build and publishing processes. - Restrict artifact publishing to the main (hyperledger/indy-plenum) repository. - Retain the uploaded deb package in the workflow for 5 days. Signed-off-by: Wade Barnes <wade@neoterictech.ca>

WadeBarnes · 2021-07-06T18:31:02Z

I found one small change needed while integrating these changes into indy-node, the signing key update. @ianco you'll recognize this one; https://github.com/hyperledger/indy-plenum/pull/1541/files#diff-1858f58aa9cdc1ba61183170b8a5916f1c6503c094b98563ca913935e3027f75R4-R6

ianco · 2021-07-06T18:39:34Z

I found one small change needed while integrating these changes into indy-node, the signing key update. @ianco you'll recognize this one; https://github.com/hyperledger/indy-plenum/pull/1541/files#diff-1858f58aa9cdc1ba61183170b8a5916f1c6503c094b98563ca913935e3027f75R4-R6

@WadeBarnes good catch!

WadeBarnes · 2021-07-06T18:41:18Z

(ci) test this please

udosson requested a review from a team as a code owner May 3, 2021 14:52

WadeBarnes self-assigned this May 10, 2021

WadeBarnes requested changes May 11, 2021

View reviewed changes

This was referenced Jul 5, 2021

Ubuntu 20.04: create CD for Ubuntu 20.04 #1537

Closed

Ubuntu 20.04: create CI for Ubuntu 20.04 #1536

Closed

publishing artifacts to jfrog artifactory

d5c7b7d

Signed-off-by: udosson <r.klemens@yahoo.de>

udosson force-pushed the publishing-artifacts branch from efe0af5 to d5c7b7d Compare July 6, 2021 09:15

WadeBarnes requested review from ianco and Toktar July 6, 2021 12:30

WadeBarnes previously approved these changes Jul 6, 2021

View reviewed changes

WadeBarnes requested review from m00sey and mirgee July 6, 2021 12:31

WadeBarnes requested a review from ryjones July 6, 2021 15:14

ryjones previously approved these changes Jul 6, 2021

View reviewed changes

m00sey reviewed Jul 6, 2021

View reviewed changes

ianco previously approved these changes Jul 6, 2021

View reviewed changes

WadeBarnes dismissed stale reviews from ianco, ryjones, and themself via fa677e7 July 6, 2021 18:29

WadeBarnes force-pushed the publishing-artifacts branch from c09b850 to fa677e7 Compare July 6, 2021 18:29

ianco approved these changes Jul 6, 2021

View reviewed changes

WadeBarnes approved these changes Jul 6, 2021

View reviewed changes

WadeBarnes merged commit c5e7a4c into hyperledger:master Jul 6, 2021

udosson mentioned this pull request Jul 8, 2021

Ubuntu 20.04: Publishing artifacts (debian packages) #1545

Merged

WadeBarnes mentioned this pull request Jul 10, 2021

Add support for publishing Debian packages hyperledger/indy-node#1680

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Publishing artifacts #1541

Publishing artifacts #1541

udosson commented May 3, 2021

sovbot commented May 3, 2021

WadeBarnes commented May 10, 2021

WadeBarnes left a comment •

edited

WadeBarnes May 11, 2021

udosson May 11, 2021

WadeBarnes May 11, 2021

WadeBarnes May 11, 2021

WadeBarnes commented Jun 16, 2021

WadeBarnes commented Jun 21, 2021

swcurran commented Jun 21, 2021

WadeBarnes commented Jun 22, 2021

WadeBarnes commented Jun 22, 2021

WadeBarnes commented Jun 22, 2021

WadeBarnes commented Jul 5, 2021

WadeBarnes commented Jul 5, 2021

udosson commented Jul 6, 2021

WadeBarnes commented Jul 6, 2021

WadeBarnes commented Jul 6, 2021

WadeBarnes commented Jul 6, 2021

ryjones left a comment

m00sey left a comment •

edited

WadeBarnes commented Jul 6, 2021

ianco left a comment

WadeBarnes commented Jul 6, 2021

ianco commented Jul 6, 2021

WadeBarnes commented Jul 6, 2021

Publishing artifacts #1541

Publishing artifacts #1541

Conversation

udosson commented May 3, 2021

sovbot commented May 3, 2021

WadeBarnes commented May 10, 2021

WadeBarnes left a comment • edited

Choose a reason for hiding this comment

WadeBarnes May 11, 2021

Choose a reason for hiding this comment

udosson May 11, 2021

Choose a reason for hiding this comment

WadeBarnes May 11, 2021

Choose a reason for hiding this comment

WadeBarnes May 11, 2021

Choose a reason for hiding this comment

WadeBarnes commented Jun 16, 2021

WadeBarnes commented Jun 21, 2021

swcurran commented Jun 21, 2021

WadeBarnes commented Jun 22, 2021

WadeBarnes commented Jun 22, 2021

WadeBarnes commented Jun 22, 2021

WadeBarnes commented Jul 5, 2021

WadeBarnes commented Jul 5, 2021

udosson commented Jul 6, 2021

WadeBarnes commented Jul 6, 2021

WadeBarnes commented Jul 6, 2021

WadeBarnes commented Jul 6, 2021

ryjones left a comment

Choose a reason for hiding this comment

m00sey left a comment • edited

Choose a reason for hiding this comment

WadeBarnes commented Jul 6, 2021

ianco left a comment

Choose a reason for hiding this comment

WadeBarnes commented Jul 6, 2021

ianco commented Jul 6, 2021

WadeBarnes commented Jul 6, 2021

WadeBarnes left a comment •

edited

m00sey left a comment •

edited