Updating the CI system #8765

General-Beck · 2018-06-02T02:05:17Z

Updating the CI system with the publication of releases and binary files on github. example
Removed i686 and armv6 builds.
Removed md5 hash.
Added parity/parity-evm Docker image.
After each binary file is builded for a given architecture, they are automatically packaged for different installers.
Artifacts are created at each stage and stored for a month on the server.
binaries from build stage
packages
List of generated systems and packages:

OS	Arch	Package
Ubuntu	arm64	deb
*	arm64	snap
*	armv7	deb
*	armv7	snap
*	amd64	deb
*	amd64	snap
Debian	amd64	deb
CentOS	x86_64	rpm
Windows	x86_64	exe (installer)
OSX	x86_64	pkg (installer)
Docker	x86_64	image

Improved build time with the caching cargo.
Need to test on nightly and tags.
We will need to change the vanity to add files to the download section.

and many more changes...
P.S. now everything has become beautiful!

…les on github Signed-off-by: Denis S. Soldatov aka General-Beck <general.beck@gmail.com>

General-Beck · 2018-06-02T02:28:31Z

trigger test

kirushik · 2018-06-04T10:45:47Z

scripts/gitlab/build.sh

@@ -1,7 +1,5 @@
 #!/bin/bash


Are we even using this build.sh monstrosity?
I think at least partially the idea was to replace it with smaller and more straightforward task-specific scripts (which it seems like you did). Also, build.sh is never mentioned in the gitlab-ci.yml

kirushik · 2018-06-04T10:55:14Z

scripts/gitlab/push-binaries.sh

+fi
+aws s3 rm --recursive s3://$S3_BUCKET/$CI_BUILD_REF_NAME/$BUILD_PLATFORM
+aws s3api put-object --bucket $S3_BUCKET --key $CI_BUILD_REF_NAME/$BUILD_PLATFORM/parity$S3WIN --body target/$PLATFORM/release/parity$S3WIN
+aws s3api put-object --bucket $S3_BUCKET --key $CI_BUILD_REF_NAME/$BUILD_PLATFORM/parity$S3WIN.md5 --body parity$S3WIN.md5


Do we even build md5 in our new pipeline?
Let's just stop doing that, and stop caring about it?
sha256 is better in every possible way, let's just stick to it?

kirushik

Overall — looks quite good; I've leaf a couple of comments on some smaller issues above.

5chdn · 2018-06-04T11:12:02Z

Okay, thanks @General-Beck. There is only one reasonable way to test this:

* [new tag]             v1.12.0-ci1 -> v1.12.0-ci1

https://gitlab.parity.io/parity/parity/pipelines/20519

5chdn · 2018-06-04T13:59:54Z

CI is all red

https://gitlab.parity.io/parity/parity/-/jobs/89154

kirushik · 2018-06-04T15:11:49Z

@General-Beck Looks like wasm-tests submodule is not updated properly for tests?

5chdn · 2018-06-05T08:03:53Z

@General-Beck is this up to date with master?

5chdn · 2018-07-02T12:20:13Z

Assuming you are working on this somewhere else (Gitlab?), I'll close this and we can reopen this once ready. Please keep me in the loop, happy to provide early feedback.

kirushik · 2018-08-21T11:41:09Z

.gitlab-ci.yml

-    - scripts/gitlab-build.sh i686-unknown-linux-gnu i686-unknown-linux-gnu i386 gcc g++ linux
-  tags:
-    - rust-i686
+    - gitlab-next


Shouldn't this be removed before the merge?

kirushik · 2018-08-21T11:43:05Z

.gitlab-ci.yml

+  only:                            &publishable_branches
+    - nightly # Our nightly builds from schedule, on `master`
+    - "v*" # Our version tags
+    - gitlab-next


Same here, seems to be a stub for testing, not something we need to merge.

kirushik · 2018-08-21T12:27:16Z

.gitlab-ci.yml

-  image: parity/rust:gitlab-ci
+    - scripts/gitlab/publish-docker.sh parity-evm
+
+publish:github:s3:


This (and the corresponding script push.sh) are named misleadingly. I can easily imagine us forgetting that we're pushing to both locations from the way this section and accompanying script are called.

Also: why don't we have pushing to S3 and pushing to Github as separate steps?

+1 for separate steps if possible

kirushik · 2018-08-21T12:57:21Z

.gitlab-ci.yml

+
+####stage:                          docs
+
+json:rpc:docs:


kirushik · 2018-08-21T13:08:38Z

scripts/gitlab/rpc-docs.sh

+
+build_docs() {
+    echo "__________Build docs__________"
+    npm install


Let's prefer yarn everywhere, shouldn't we?
Dependencies caching should become much more straightforward with it.

kirushik · 2018-08-21T18:28:41Z

scripts/gitlab/rpc-docs.sh

+commit_files() {
+    echo "__________Commit files__________"
+    git checkout -b rpcdoc-update-${CI_COMMIT_REF_NAME}
+    git commit .


Isn't it intended to be git add here?

5chdn

LGMT, only minor grumbles. I'm in favor of merging this early and break stuff rather than keeping this around much longer :)

5chdn · 2018-08-21T18:50:20Z

.gitlab-ci.yml

-    - scripts/gitlab-build.sh i686-unknown-linux-gnu i686-unknown-linux-gnu i386 gcc g++ linux
-  tags:
-    - rust-i686
+    - gitlab-next


5chdn · 2018-08-21T18:52:01Z

.gitlab-ci.yml

+.publishable_branches:             # list of git refs for publishing builds to the "production" locations
+  only:                            &publishable_branches
+    - nightly # Our nightly builds from schedule, on `master`
+    - "v*" # Our version tags


branch vitalik-improvements would match this, right? what about "v2*" or even regex if possible?

Great catch!

5chdn · 2018-08-21T18:55:32Z

.gitlab-ci.yml

-    - tags
-    - stable
-    - triggers
+    - master


does this mean tests are not run on pull requests anymore?

Only tests with <<: *optional_test included; testing with stable happens everywhere (see section test:rust:stable:); testing with beta and nightly Rust versions are only limited to master branch. It makes some sense, but not the perfect one — probably we want to be sure that our code breaks on nightly immediately, not during the next release.

5chdn · 2018-08-21T18:57:10Z

.gitlab-ci.yml

+  variables:
+    CARGO_TARGET:                  armv7-linux-androideabi
+
+build:macos:x86_64:


for consistency, can we rename this to build:darwin:macos:x86_64?

5chdn · 2018-08-21T19:00:44Z

.gitlab-ci.yml

-windows:
+  <<:                              *collect_artifacts
+
+build-windows-x86_64:


also build:windows:msvc:x86_64?

Unfortunately, windows builds can't have : characters in their names, because windows paths can't have said character.

Maybe let's come up with a better (and universal) separator?

Ok let's go back to - then

5chdn · 2018-08-21T19:04:32Z

.gitlab-ci.yml

-  image: parity/rust:gitlab-ci
+    - scripts/gitlab/publish-docker.sh parity-evm
+
+publish:github:s3:


+1 for separate steps if possible

5chdn · 2018-08-21T19:10:00Z

scripts/gitlab/install-readme.sh

+
+If you continue, Parity will be installed as a user service. You will be able to use the Parity Wallet through your browser by using the menu bar icon, following the shortcut in the Launchpad or navigating to http://localhost:8180/ in your browser.
+
+Parity is distributed under the terms of the GPL."


What is this?

5chdn · 2018-08-21T19:11:07Z

scripts/gitlab/publish-snap.sh

+set -u # treat unset variables as error
+
+case ${CI_COMMIT_REF_NAME} in
+  master|*v2.1*|gitlab-next) export CHANNEL="edge";;


replace gitlab-next with nightly?

maybe it's sufficient on nightly only, not master

5chdn · 2018-08-21T19:13:33Z

scripts/gitlab/rpc-docs.sh

+commit_files() {
+    echo "__________Commit files__________"
+    git checkout -b rpcdoc-update-${CI_COMMIT_REF_NAME}
+    git commit .


5chdn · 2018-08-21T19:14:22Z

scripts/gitlab/rpc-docs.sh

+
+upload_files() {
+    echo "__________Upload files__________"
+    git push --tags


Does this include #9219 ?

gabreal

good job, let's give it a try.

gabreal · 2018-08-22T13:29:27Z

docker/hub/Dockerfile

- curl https://sh.rustup.rs -sSf | sh -s -- -y && \
-# rustup directory
- PATH=/root/.cargo/bin:$PATH && \
+RUN apt update && apt install -y --no-install-recommends openssl libudev-dev file


Why libudev-dev? Shouldn't it be libudev?
And there is no dependency on libssl anymore, so why openssl?

gabreal · 2018-08-22T13:43:48Z

scripts/gitlab/templates/snapcraft.template.yaml

+      cp -v ethkey $SNAPCRAFT_PART_INSTALL/usr/bin/ethkey
+      cp -v ethstore $SNAPCRAFT_PART_INSTALL/usr/bin/ethstore
+      cp -v whisper $SNAPCRAFT_PART_INSTALL/usr/bin/whisper
+    stage-packages: [libc6, libssl1.0.0, libudev1, libstdc++6, cmake]


https://docs.snapcraft.io/build-snaps/syntax#stage-packages
so the files from these packages end up in the snap. if this is required it should be the same for the docker image.

gabreal · 2018-08-22T15:46:54Z

.gitlab-ci.yml

-    - parity.zip
-    name: "x86_64-apple-darwin_parity"
-windows:
+  <<:                              *collect_artifacts


the previous jobs (e.g. linux:linux:android:armhf) get their *collect_artifacts block from the *build anchor, doesn't that also work for this one?

5chdn · 2018-08-24T20:15:37Z

Addressed most of the issues. Will merge once the CI is green. And then we can work from there.

5chdn · 2018-08-24T20:58:57Z

LGTM https://gitlab.parity.io/parity/parity/pipelines/22683

kirushik

@5chdn Will this master→nightly change do what you actually need? If yes, please merge at will.

kirushik · 2018-08-24T22:23:27Z

.gitlab-ci.yml

  <<:                              *publish_docker
  script:
    - scripts/gitlab/publish-docker.sh parity-evm

-publish:github:s3:
+publish-github-and-s3:


In my ideal world we would split this into two separate steps.
But let's merge it as it is for now.

kirushik · 2018-08-24T22:24:23Z

scripts/gitlab/package-snap.sh

@@ -3,7 +3,7 @@
 set -e # fail on any error
 set -u # treat unset variables as error
 case ${CI_COMMIT_REF_NAME} in
-  master|*v2.1*|gitlab-next) export GRADE="devel";;
+  nightly|*v2.1*) export GRADE="devel";;


Are we going to publish nightlies only from the special nightly branch? Why not from the master?

master is triggered always after we merge a pull request, so multiple times per day and nightly is only happing once per night which is sufficient. now, master does not create a snap which is fine IMHO.

@5chdn Alternatively, we can avoid triggering the actual build-and-package on master on every push (only tests), and trigger the builds only once per day with a cron-like trigger (Gitlab CI already supports those). This way, there would be no need in maintaining a separate nightly branch manually.

If you agree with me that not having to maintain separate nightly is desired, please open a separate issue for this (or let me know, and I'll do this).

we are doing that. nightly is a tag.

5chdn · 2018-08-25T22:44:19Z

🎉

* master: evmbin: Fix gas_used issue in state root mismatch and handle output better (#9418) Update hardcoded sync (#9421) Add block reward contract config to ethash and allow off-chain contracts (#9312) Private packets verification and queue refactoring (#8715) Update tobalaba.json (#9419) docs: add parity ethereum logo to readme (#9415) build: update rocksdb crate (#9414) Updating the CI system (#8765) Better support for eth_getLogs in light mode (#9186) Add update docs script to CI (#9219) `gasleft` extern implemented for WASM runtime (kip-6) (#9357) block view! removal in progress (#9397) Prevent sync restart if import queue full (#9381) nonroot CentOS Docker image (#9280) ethcore: kovan: delay activation of strict score validation (#9406)

Updating the CI system with the publication of releases and binary fi…

ebc6b9c

…les on github Signed-off-by: Denis S. Soldatov aka General-Beck <general.beck@gmail.com>

General-Beck added A0-pleasereview 🤓 Pull request needs code review. M0-build 🏗 Building and build system. M1-ci 🙉 Continuous integration. M2-installer 📲 Installers for MacOS and Windows. labels Jun 2, 2018

General-Beck self-assigned this Jun 2, 2018

General-Beck requested review from kirushik, jesuscript, tomaka and 5chdn June 2, 2018 02:05

General-Beck added 3 commits June 2, 2018 05:13

add missed scripts

8926b43

chmod +x scripts

95a7b03

fix download link for github

514e4af

5chdn added this to the 1.12 milestone Jun 2, 2018

kirushik reviewed Jun 4, 2018

View reviewed changes

5chdn added the A4-awaitingci 🤖 Pull request is waiting for changes on the CI to complete tests before review/merge can begin. label Jun 4, 2018

5chdn mentioned this pull request Jun 18, 2018

Remove debian/.deb and centos/.rpm packaging scripts #8887

Merged

5chdn closed this Jul 2, 2018

Merge branch 'master' into gitlab-next

42a0ba5

5chdn removed A4-awaitingci 🤖 Pull request is waiting for changes on the CI to complete tests before review/merge can begin. A4-gotissues 💥 Pull request is reviewed and has significant issues which must be addressed. labels Aug 20, 2018

kirushik reviewed Aug 21, 2018

View reviewed changes

5chdn approved these changes Aug 21, 2018

View reviewed changes

5chdn modified the milestones: 2.0 Ethereum, 2.1 Aug 21, 2018

5chdn added A6-mustntgrumble 💦 Pull request has areas for improvement. The author need not address them before merging. and removed A0-pleasereview 🤓 Pull request needs code review. labels Aug 21, 2018

gabreal approved these changes Aug 22, 2018

View reviewed changes

5chdn mentioned this pull request Aug 24, 2018

Add update docs script to CI #9219

Merged

5chdn added 7 commits August 24, 2018 15:51

ci: merge master

200a826

ci: fix merge leftovers

44bf39d

ci: remove gitlab-next from ci triggers

46622d2

ci: fix git add in docs script

8a0c089

ci: use nightly instead of master for publish triggers

a8c2b29

ci: remove sleep from gitlab config

c943b82

ci: replace ':' with '-' in gitlab targets

13594c5

kirushik approved these changes Aug 24, 2018

View reviewed changes

5chdn added A8-looksgood 🦄 Pull request is reviewed well. and removed A6-mustntgrumble 💦 Pull request has areas for improvement. The author need not address them before merging. labels Aug 25, 2018

ci: fix recursive copy in docs script

f1f8362

5chdn merged commit bd3bc5c into master Aug 25, 2018

5chdn deleted the gitlab-next branch August 25, 2018 22:44

andresilva mentioned this pull request Aug 30, 2018

ci: only include local paths in coverage script (except target) #9437

Merged

ordian mentioned this pull request Sep 10, 2018

ci: fix rpc docs generation #9515

Merged

5chdn mentioned this pull request Sep 29, 2018

Calculate sha3 instead of sha256 for push-release. #9673

Merged


		If you continue, Parity will be installed as a user service. You will be able to use the Parity Wallet through your browser by using the menu bar icon, following the shortcut in the Launchpad or navigating to http://localhost:8180/ in your browser.

		Parity is distributed under the terms of the GPL."


		####stage: docs

		json:rpc:docs:

Updating the CI system #8765

Updating the CI system #8765

Conversation

General-Beck commented Jun 2, 2018 • edited

General-Beck commented Jun 2, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kirushik left a comment

Choose a reason for hiding this comment

5chdn commented Jun 4, 2018

5chdn commented Jun 4, 2018

kirushik commented Jun 4, 2018 • edited

5chdn commented Jun 5, 2018

5chdn commented Jul 2, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

5chdn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gabreal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

5chdn commented Aug 24, 2018

5chdn commented Aug 24, 2018

kirushik left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

5chdn commented Aug 25, 2018

General-Beck commented Jun 2, 2018 •

edited

kirushik commented Jun 4, 2018 •

edited