Skip to content
This repository has been archived by the owner. It is now read-only.

Deploy fat jar to Azure via Travis on tags #39

Merged
merged 2 commits into from Jun 28, 2017

Conversation

Projects
None yet
3 participants
@c-w
Copy link
Member

commented Jun 27, 2017

After merging this pull request, whenever we push a tag to Github, Travis will as part of the CI automatically build a fat jar that contains project-fortis-spark and all of its dependencies and then upload that jar to a public Azure Blob storage from where our cluster launching script can pick up the jar and pass it to spark-submit.

Sample run:

This process takes about 5 minutes to run, so we could even consider running it on every Travis run (not just on tags) and name the created jars by their commit hash. This is a trivial one-line change to the .travis/publish.sh script so just let me know if you'd like that option.

Another thing that we can implement is to upload a second copy of the jar to a name like fortis-latest.jar so that the deployment script can always pick up the latest jar without having to be modified. Thoughts?

Resolves #31

@c-w c-w added the in progress label Jun 27, 2017

@jcjimenez
Copy link
Contributor

left a comment

LGTM with questions.

check_preconditions() {
if [ -z "${tag}" ]; then
log "Build is not a tag, skipping publish"
exit 0

This comment has been minimized.

Copy link
@jcjimenez

jcjimenez Jun 28, 2017

Contributor

Excellent!

This comment has been minimized.

Copy link
@c-w

c-w Jun 28, 2017

Author Member

What's your opinion on creating a fat jar for every commit instead of just tags?

This comment has been minimized.

Copy link
@jcjimenez

jcjimenez Jun 28, 2017

Contributor

I want to say I prefer tags, but if we do it automatically, we may have to navigate through a bunch of junk tags in order to promote one for uploading. By the smae token, we may also end up uploading a lot of unnecessary jars. Something else we could do is to have a release branch and we merge to it when we want to upload.

This comment has been minimized.

Copy link
@c-w

c-w Jun 28, 2017

Author Member

Alright, let's stick with the tags then :) Thanks for your input.

This comment has been minimized.

Copy link
@erikschlegel

erikschlegel Jun 28, 2017

Contributor

Not sure if you considered this, but shouldnt we create a release in GH https://developer.github.com/v3/repos/releases/#create-a-release?

This comment has been minimized.

Copy link
@c-w

c-w Jun 28, 2017

Author Member

As far as I know, all that the Github releases button does is to create a tag. This is why the test tag that I created for this PR shows up as a release:

image

As such, we can do either (create the release on Github or a tag on the git shell) and Travis will pick it up.

This comment has been minimized.

Copy link
@c-w

c-w Jun 28, 2017

Author Member

Just confirmed that this works. Here is the build for a release that I kicked off via the Github UI.

fi
if [ -z "${blobaccount}" ] || [ -z "${blobkey}" ] || [ -z "${blobcontainer}" ]; then
log "Azure blob connection is not set, unable to publish builds"
exit 1

This comment has been minimized.

Copy link
@jcjimenez

jcjimenez Jun 28, 2017

Contributor

Since this is runs after success, these non-zero exit codes won't block the PR, right?

This comment has been minimized.

Copy link
@c-w

c-w Jun 28, 2017

Author Member

Correct.

--account-key "${blobkey}" \
--file "${fatjar}" \
--container "${blobcontainer}" \
--blob "${blobname}"

This comment has been minimized.

Copy link
@jcjimenez

jcjimenez Jun 28, 2017

Contributor

Any chance we could upload to blob storage via openssl + curl so we don't have to install the cli and npm?

This comment has been minimized.

Copy link
@c-w

c-w Jun 28, 2017

Author Member

Unfortunately it's non-trivial to talk to Azure via curl (see this StackOverflow post) so I don't think it's worth it.

@erikschlegel
Copy link
Contributor

left a comment

This looks great @c-w . Left a couple comments below.

Thanks!

curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -
sudo apt-get update
sudo apt-get install -y -qq nodejs
sudo npm install -g npm

This comment has been minimized.

Copy link
@erikschlegel

erikschlegel Jun 28, 2017

Contributor

Couldnt we minimize the build time by installing the azure-cli apt-get package as opposed to installing node? i.e. sudo apt-get -y update && sudo apt-get install -y azure-cli

This comment has been minimized.

Copy link
@c-w

c-w Jun 28, 2017

Author Member

I tried a number of ways to install the azure-cli (e.g. via apt-get or the v2) and this is the only way that I got to work reliably. The apt-get package installs a really old version of node and npm which breaks some functionality downstream.

check_preconditions() {
if [ -z "${tag}" ]; then
log "Build is not a tag, skipping publish"
exit 0

This comment has been minimized.

Copy link
@erikschlegel

erikschlegel Jun 28, 2017

Contributor

Not sure if you considered this, but shouldnt we create a release in GH https://developer.github.com/v3/repos/releases/#create-a-release?

@c-w c-w merged commit b0259f7 into master Jun 28, 2017

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@c-w c-w deleted the travis-deploy branch Jun 28, 2017

@c-w c-w removed the in progress label Jun 28, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.