Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MYST-643 Make CI parallel building #287

Merged
merged 8 commits into from Aug 21, 2018

Conversation

Projects
None yet
4 participants
@tadovas
Copy link
Member

commented Jul 11, 2018

Split checks into parallel builds on test phase
Possibly run e2e and packaging in compile phase (or integrity checking)
Make deployment phase with parallel deployments of the same binaries (provided in previous phase)

@tadovas tadovas requested review from donce and Waldz as code owners Jul 11, 2018

@tadovas tadovas force-pushed the experiment/travis-stages branch 2 times, most recently from d5aad36 to 256f11b Jul 11, 2018

.travis.yml Outdated
- script: bin/test
#e2e stage
- stage: e2e-test
script: bin/travis_scripts/check_docker.sh && bin/travis_scripts/check_docker_compose.sh && bin/test_e2e

This comment has been minimized.

Copy link
@Waldz

Waldz Aug 13, 2018

Member

Not much value of those checks, drop them maybe?
Could they've be in some before_* phase?

@@ -17,25 +17,42 @@ env:
cache:
directories:
- "$CACHE_ROOT_DIR"
- vendor

This comment has been minimized.

Copy link
@Waldz

Waldz Aug 13, 2018

Member

Why we cache it, but then delete it?

This comment has been minimized.

Copy link
@soffokl

soffokl Aug 15, 2018

Member

Vendor in the cache is used here to pass it between stages, to avoid multiple glide install calls. Deleting it everytime is not required.

.travis.yml Outdated
- stage: build
script:
- bin/package_all $BUILD_VERSION linux
- bin/s3 cp build/package/mysterium_server_linux_amd64.tar.gz s3://build-artifacts

This comment has been minimized.

Copy link
@Waldz

Waldz Aug 15, 2018

Member

I suggest to copy all directory build/package which is output of script package_all. Will be more simple

.travis.yml Outdated
@@ -13,69 +13,140 @@ env:
- BUILD_BRANCH=$TRAVIS_BRANCH
- BUILD_NUMBER=$TRAVIS_BUILD_NUMBER
- BUILD_VERSION_MAJOR=0
- PATH=$HOME/.local/bin:$PATH

This comment has been minimized.

Copy link
@Waldz

Waldz Aug 15, 2018

Member

That binaries are here? Maybe comment would help or self-describing VARIABLE=$HOME/.local/bin

.travis.yml Outdated
name: "Linux packages"

- script:
- bin/package_all $BUILD_VERSION windows

This comment has been minimized.

Copy link
@Waldz

Waldz Aug 15, 2018

Member

Lets depreciate script package_all with tis PR and lets call not use it as proxy script.
Each job calls specific command it needs e.g. BINARY=build/server/mysterium_server_windows_amd64.exe bin/server_package_standalone windows

.travis.yml Outdated
- bin/package_all $BUILD_VERSION linux
- bin/s3 cp build/package/mysterium_server_linux_amd64.tar.gz s3://build-artifacts
- bin/s3 cp build/package/mysterium_client_linux_amd64.tar.gz s3://build-artifacts
- bin/s3 cp build/package/mysterium-client_linux_amd64.deb s3://build-artifacts

This comment has been minimized.

Copy link
@Waldz

Waldz Aug 15, 2018

Member

Would be nice to have Debian build in separate job also

.travis.yml Outdated
name: "Ubuntu docker image for client"

#deploy artifacts from master
- stage: deploy-master

This comment has been minimized.

Copy link
@Waldz

Waldz Aug 15, 2018

Member

rename suggestion: deploy-master -> release-dev

- script:
- bin/client_package_docker_ubuntu $BUILD_VERSION
- docker save mysterium-client:ubuntu | gzip > mysterium-client_ubuntu.tgz
- bin/s3 cp mysterium-client_ubuntu.tgz s3://docker-images

This comment has been minimized.

Copy link
@Waldz

Waldz Aug 15, 2018

Member

+1 wow, exporting image without docker registry

This comment has been minimized.

Copy link
@soffokl

soffokl Aug 16, 2018

Member

The idea was to avoid pushing other images if at least one failed since it could be incompatible. So push to docker registry is done on the next stage.

.travis.yml Outdated
- bin/s3 cp mysterium-client_ubuntu.tgz s3://docker-images
name: "Ubuntu docker image for client"

#deploy artifacts from master

This comment has been minimized.

Copy link
@Waldz

Waldz Aug 15, 2018

Member

# Development bleeding-edge release (on master commits only)

.travis.yml Outdated
- bin/release_docker dev
name: "Pushing dev docker images"

#deploy stage

This comment has been minimized.

Copy link
@Waldz

Waldz Aug 15, 2018

Member

# Official release (on tags only)

@Waldz Waldz changed the title WIP: Experiment with travis stages and parallel jobs MYST-643 Experiment with travis stages and parallel jobs Aug 15, 2018

@Waldz Waldz changed the title MYST-643 Experiment with travis stages and parallel jobs MYST-643 Make CI parallel building Aug 15, 2018

name: "Formating check"

- script: bin/check_license
name: "License check"

This comment has been minimized.

Copy link
@tadovas

tadovas Aug 17, 2018

Author Member

These sanity checkers like license, linter, vet are very small jobs (few sec), is there a point to paralelize them - overhead of single job on travis is around 30 secs, and there is limited number of parallel jobs.

This comment has been minimized.

Copy link
@soffokl

soffokl Aug 17, 2018

Member

I did a test, the first test was with license+vet+fmt in one job, and the second test with all steps in separate jobs. Got the same results for the whole stage: 1:54 for both cases.
If we are starting all jobs in parallel, the time of the stage will be equal to the longest job.
We still need to wait for other jobs to finish, like unit tests.
And, I think, separated jobs looks more clear in the Travis CI UI.

This comment has been minimized.

Copy link
@tadovas

tadovas Aug 17, 2018

Author Member

💯 karma points for making tests and measurements. And yes - I was thinking between clear separations and a bit of optimisation. Looks good to me now

@soffokl soffokl dismissed stale reviews from zolia and Waldz via d96b8cb Aug 20, 2018

@soffokl

This comment has been minimized.

Copy link
Member

commented Aug 20, 2018

@zolia @Waldz Could you please review this PR once again. I have moved E2E-tests to the early stage to finish it faster.

@soffokl soffokl force-pushed the experiment/travis-stages branch from d96b8cb to 3969f99 Aug 21, 2018

tadovas and others added some commits Jul 11, 2018

@soffokl soffokl dismissed stale reviews from zolia and Waldz via e7acbbc Aug 21, 2018

@soffokl soffokl force-pushed the experiment/travis-stages branch from 3969f99 to e7acbbc Aug 21, 2018

@zolia

zolia approved these changes Aug 21, 2018

@Waldz

Waldz approved these changes Aug 21, 2018

@tadovas tadovas merged commit f652ff3 into master Aug 21, 2018

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@tadovas tadovas deleted the experiment/travis-stages branch Aug 21, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.