-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BEAM-1376] Move website jobs to main Beam repo to reduce duplication. #1897
Changes from all commits
d87b1dd
b31f8cd
1a96085
0699dcc
759455c
3c383fe
47cca8d
e30a4c0
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,62 @@ | ||
import common_job_properties | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This file could logically stay there too. Is it feasible? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not feasible without doing ugly things inside the seed job. Seed job builds by cloning this repo and running all jobs inside the .jenkins folder -- we could hypothetically also manually clone the website repo, have its groovy jobs reference (essentially) ../../beam/.jenkins/common... but that seems to me to be a fairly brittle and undesirable solution. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I've had bad experiences with submodules relevant to the article I posted there -- if you really want to use them I'll give it a shot, but I think the downsides are way larger than having these two (eventually three) files live in a different repository. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. My experience matches too. Another alternative to consider: another Jenkins step before the DSL step that clones the other repository? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah I'd thought of that but I'm pretty resistant to having the jobs be dependent on the configuration of their environment to succeed -- if something goes wrong with the clone it could adversely affect the jobs running in Jenkins. Having all of our jenkins jobs defined in one place seems like a better choice. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Alright -- stays as is. |
||
|
||
// Defines a job. | ||
job('beam_PreCommit_Website_Stage') { | ||
description('Stages the pull requests proposed for the Apache Beam ' + | ||
'website to a temporary location to ease reviews.') | ||
|
||
// Set common parameters. | ||
common_job_properties.setTopLevelWebsiteJobProperties(delegate) | ||
|
||
// Set pull request build trigger. | ||
common_job_properties.setPreCommit( | ||
delegate, | ||
'Jenkins: automatic staging of pull requests', | ||
'\nJenkins built the site at commit id ${ghprbActualCommit} with ' + | ||
'Jekyll and staged it [here](http://apache-beam-website-pull-' + | ||
'requests.storage.googleapis.com/${ghprbPullId}/index.html). ' + | ||
'Happy reviewing.\n\nNote that any previous site has been deleted. ' + | ||
'This staged site will be automatically deleted after its TTL ' + | ||
'expires. Push any commit to the pull request branch or re-trigger ' + | ||
'the build to get it staged again.') | ||
|
||
steps { | ||
// Run the following shell script as a build step. | ||
shell ''' | ||
# Install RVM. | ||
gpg --keyserver hkp://keys.gnupg.net --recv-keys \\ | ||
409B6B1796C275462A1703113804BB82D39DC0E3 | ||
\\curl -sSL https://get.rvm.io | bash | ||
source /home/jenkins/.rvm/scripts/rvm | ||
|
||
# Install Ruby. | ||
RUBY_VERSION_NUM=2.3.0 | ||
rvm install ruby $RUBY_VERSION_NUM --autolibs=read-only | ||
|
||
# Install Bundler gem | ||
PATH=~/.gem/ruby/$RUBY_VERSION_NUM/bin:$PATH | ||
GEM_PATH=~/.gem/ruby/$RUBY_VERSION_NUM/:$GEM_PATH | ||
gem install bundler --user-install | ||
|
||
# Install all needed gems. | ||
bundle install --path ~/.gem/ | ||
|
||
# Remove current site if it exists. | ||
GCS_PATH="gs://apache-beam-website-pull-requests/${ghprbPullId}/" | ||
gsutil -m rm -r -f ${GCS_PATH} || true | ||
|
||
# Build the new site with the baseurl specified. | ||
rm -fr ./content/ | ||
bundle exec jekyll build --baseurl=/${ghprbPullId} | ||
|
||
# Install BeautifulSoup HTML Parser for python. | ||
pip install --user beautifulsoup4 | ||
|
||
# Fix links on staged website. | ||
python .jenkins/append_index_html_to_internal_links.py | ||
|
||
# Upload the new site. | ||
gsutil -m cp -R ./content/* ${GCS_PATH} | ||
'''.stripIndent().trim() | ||
} | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,47 @@ | ||
import common_job_properties | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same as above. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. See above. |
||
|
||
// Defines a job. | ||
job('beam_PreCommit_Website_Test') { | ||
description('Runs tests on the pull requests proposed for the Apache Beam ' + | ||
'website.') | ||
|
||
// Set common parameters. | ||
common_job_properties.setTopLevelWebsiteJobProperties(delegate) | ||
|
||
// Execute concurrent builds. Multiple builds of this project may be executed | ||
// in parallel. This is safe because this build does not require exclusive | ||
// access to any shared resources. | ||
concurrentBuild() | ||
|
||
// Set pull request build trigger. | ||
common_job_properties.setPreCommit( | ||
delegate, | ||
'Jenkins: test website (dead links, etc.)') | ||
|
||
steps { | ||
// Run the following shell script as a build step. | ||
shell ''' | ||
# Install RVM. | ||
gpg --keyserver hkp://keys.gnupg.net --recv-keys \\ | ||
409B6B1796C275462A1703113804BB82D39DC0E3 | ||
\\curl -sSL https://get.rvm.io | bash | ||
source /home/jenkins/.rvm/scripts/rvm | ||
|
||
# Install Ruby. | ||
RUBY_VERSION_NUM=2.3.0 | ||
rvm install ruby $RUBY_VERSION_NUM --autolibs=read-only | ||
|
||
# Install Bundler gem | ||
PATH=~/.gem/ruby/$RUBY_VERSION_NUM/bin:$PATH | ||
GEM_PATH=~/.gem/ruby/$RUBY_VERSION_NUM/:$GEM_PATH | ||
gem install bundler --user-install | ||
|
||
# Install all needed gems. | ||
bundle install --path ~/.gem/ | ||
|
||
# Build the new site and test it. | ||
rm -fr ./content/ | ||
bundle exec rake test | ||
'''.stripIndent().trim() | ||
} | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -19,13 +19,14 @@ | |
import common_job_properties | ||
|
||
// Defines the seed job, which creates or updates all other Jenkins projects. | ||
job('beam_SeedJob_Main') { | ||
description('Automatically configures all Apache Beam main repo Jenkins ' + | ||
'projects based on Jenkins DSL groovy files checked into the ' + | ||
'code repository.') | ||
job('beam_SeedJob') { | ||
description('Automatically configures all Apache Beam Jenkins projects based' + | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. space between "based" and "on" There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done |
||
' on Jenkins DSL groovy files checked into the code repository.') | ||
|
||
previousNames('beam_SeedJob_Main') | ||
|
||
// Set common parameters. | ||
common_job_properties.setTopLevelJobProperties(delegate) | ||
common_job_properties.setTopLevelMainJobProperties(delegate) | ||
|
||
// This is a post-commit job that runs once per day, not for every push. | ||
common_job_properties.setPostCommit( | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a continuation of the sentence on the above line -- capitalizing seems wrong here?