-
Notifications
You must be signed in to change notification settings - Fork 26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improvements to Jenkinsfile #147
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some thoughts
Jenkinsfile
Outdated
sh 'cd config;. venv/bin/activate; pigz merged-kg.tar' | ||
dir('./gitrepo') { | ||
sh '. venv/bin/activate; python3.7 run.py load' | ||
sh '. venv/bin/activate; pigz merged-kg.tar' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why does this need activate?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
run.py load
needs the venv environment - I guess pigz
doesn't though
Jenkinsfile
Outdated
sh 'cd config;. venv/bin/activate; pigz merged-kg.nt' | ||
dir('./gitrepo') { | ||
sh '. venv/bin/activate; kgx transform --input-type tsv --output-type nt -o ./merged-kg.nt merged-kg.tar.gz' | ||
sh '. venv/bin/activate; pigz merged-kg.nt' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When is activate used?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
all the run.py
commands will need this, possibly kgx too. Looks like I can remove it from the pigz command though
Jenkinsfile
Outdated
if (run_py_dl == 0) { // upload raw to s3 | ||
sh 's3cmd -c $S3CMD_JSON --acl-public --mime-type=plain/text --cf-invalidate put -r data/raw s3://kg-hub-public-data/' | ||
} else { // 'run.py download' failed - let's try to download last good copy of raw/ from s3 to data/ | ||
sh 'rm -fr data/raw; mkdir -p data/raw' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I might suggest:
'rm -r -f data/raw || true'
'mkdir -p data/raw || true'
or functionally similar
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks
Jenkinsfile
Outdated
sh 'cd config;. venv/bin/activate; python3.7 run.py load' | ||
sh 'cd config;. venv/bin/activate; pigz merged-kg.tar' | ||
dir('./gitrepo') { | ||
sh '. venv/bin/activate; python3.7 run.py load' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe '&&' for these?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
probably a good idea
cd gitrepo