Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SNOW-964536 benchmark setup #655

Merged
merged 26 commits into from
Dec 12, 2023
Merged
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
4d78d77
SNOW-964536 Add pipeline Jenkinsfile
sfc-gh-lthiede Nov 14, 2023
2a9fa60
SNOW-964536 Test change to trigger pipeline
sfc-gh-lthiede Nov 15, 2023
ec48772
SNOW-964536 Test accessing passed parameters
sfc-gh-lthiede Nov 15, 2023
3f0dcf6
SNOW-964536 Try determining tag and try cloning benchmark setup
sfc-gh-lthiede Nov 15, 2023
6418eb1
SNOW-964536 Checkout benchmark setup without unavailable import
sfc-gh-lthiede Nov 15, 2023
95f826d
SNOW-964536 FIx command for determining git tag
sfc-gh-lthiede Nov 15, 2023
8a7b341
SNOW-964536 Run setup on one deployment given by parameters
sfc-gh-lthiede Nov 15, 2023
995a673
SNOW-964536 Fix variable definition
sfc-gh-lthiede Nov 15, 2023
7d5ceb1
SNOW-964536 Test if username and password for accessing deployments a…
sfc-gh-lthiede Nov 15, 2023
299961f
SNOW-964536 Change to trigger pipeline
sfc-gh-lthiede Nov 15, 2023
93ac3e3
SNOW-964536 Run setup script
sfc-gh-lthiede Nov 17, 2023
bb89b13
SNOW-964536 Push to trigger pipeline
sfc-gh-lthiede Nov 17, 2023
002ac5d
SNOW-964536 Debug problem with decrypting config file
sfc-gh-lthiede Nov 17, 2023
0d60e7a
SNOW-964536 Clean up artifacts from last run before starting new run
sfc-gh-lthiede Nov 17, 2023
b33c9e6
SNOW-964536 Print debug info
sfc-gh-lthiede Nov 17, 2023
307dc17
SNOW-964536 Only cleanup directory of setup application before clonin…
sfc-gh-lthiede Nov 17, 2023
e08972d
SNOW-964536 Fix usage of deleteDir
sfc-gh-lthiede Nov 17, 2023
c69fd8f
SNOW-964536 Test marking schema as complete after insert
sfc-gh-lthiede Nov 17, 2023
4dd0c84
SNOW-964536 Try running setup for multiple deployments in loop
sfc-gh-lthiede Nov 17, 2023
7661540
SNOW-964536 Fix variable name
sfc-gh-lthiede Nov 17, 2023
fecc1e0
SNOW-964536 Trigger independent jobs to run setups on different deplo…
sfc-gh-lthiede Nov 20, 2023
abe889a
SNOW-964536 Run pipeline in Jenkinsfile
sfc-gh-lthiede Nov 20, 2023
ba8d9a1
SNOW-964536 Derive valid database name from tag
sfc-gh-lthiede Nov 22, 2023
f2ec4ae
SNOW-964536 Print database name friendly tag to debug error
sfc-gh-lthiede Nov 22, 2023
493e050
SNOW-964536 Escape dot in split regex
sfc-gh-lthiede Nov 22, 2023
efb08bf
SNOW-964536 Setup both Preprod12 and QA3
sfc-gh-lthiede Dec 7, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions Jenkinsfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
pipeline {
agent { label 'parallelizable-c7' }
options { timestamps() }
environment {
ingest_sdk_dir = "${WORKSPACE}/snowflake-ingest-java"
ingest_sdk_tag = sh(returnStdout: true, script: "cd $ingest_sdk_dir && git describe --tags").trim()

}
stages {
stage('TriggerJobs') {
steps {
script {
def valid_db_name_tag = ingest_sdk_tag.split('\\.').join('_')
def deployments = [
"qa3": {
build job: "SFPerf-Other-Jobs/TPCDS_BDEC_Setup",
parameters: [
string(name: 'ingest_sdk_github_branch', value: ingest_sdk_tag),
string(name: 'database', value: "STREAMING_INGEST_BENCHMARK_DB_${valid_db_name_tag}"),
string(name: 'deployment', value: 'qa3.us-west-2.aws'),
string(name: 'tpcds_scale_factor', value: 'sf1000')
],
propagate: true
},
"preprod12": {
build job: "SFPerf-Other-Jobs/TPCDS_BDEC_Setup",
parameters: [
string(name: 'ingest_sdk_github_branch', value: ingest_sdk_tag),
string(name: 'database', value: "STREAMING_INGEST_BENCHMARK_DB_${valid_db_name_tag}"),
string(name: 'deployment', value: 'preprod12.us-west-2.aws'),
string(name: 'tpcds_scale_factor', value: 'sf1000')
],
propagate: true
}
]

parallel deployments
}
}
}
}
}
Loading