Delete cluster after test timeout. #6437
Conversation
Summary: So far Jenkins kills the system test job after two hours. This also kills our cleanup call. This patch introduces a bash timeout that still leaves time for a cluster cleanup. JIRA issues: MARATHON-8364
There was a problem hiding this comment.
I'm building your change at jenkins-marathon-pipelines-PR-6437-1.
There was a problem hiding this comment.
✔ Build of #6437 completed successfully.
See details at jenkins-marathon-pipelines-PR-6437-1.
You can create a DC/OS with your patched Marathon by creating a new pull
request with the following changes in buildinfo.json:
"url": "https://s3.amazonaws.com/downloads.mesosphere.io/marathon/builds/1.7.53-c69de5faf/marathon-1.7.53-c69de5faf.tgz",
"sha1": "31a5bf25971ca939fe9de3bfa424d6e12ef7324f"You can run system integration test changes of this PR against Marathon
master by triggering this Jenkins job with the Pull_Request_id 6437.
The job then reports back to this PR.
＼\ ٩( ᐛ )و /／
|
Validate in build https://jenkins.mesosphere.com/service/jenkins/job/system-integration-tests/job/marathon-si-pr/237/.
|
ci/si_install_deps.sh
Outdated
| @@ -22,5 +22,10 @@ if ! command -v amm >/dev/null 2>&1; then | |||
| echo "Ammonite successfully installed" | |||
| fi | |||
|
|
|||
| # Ensure timeout is available. | |||
| if [ "$PLATFORM" == 'Darwin' ]; then | |||
There was a problem hiding this comment.
One has to have brew install coreutils but ok.
ci/si_pipeline.sh
Outdated
| @@ -76,7 +76,7 @@ case $CLUSTER_LAUNCH_CODE in | |||
| 0) | |||
| "$ROOT_PATH/ci/dataDogClient.sc" "marathon.build.si.$VARIANT.cluster_launch.success" 1 | |||
| cp -f "$DOT_SHAKEDOWN" "$HOME/.shakedown" | |||
| make test | |||
| timeout --preserve-status -s KILL 1s make test | |||
There was a problem hiding this comment.
so we're killing after 1s? 🤣 I don't think we're that fast yet
There was a problem hiding this comment.
We are getting there...
There was a problem hiding this comment.
I'm building your change at jenkins-marathon-pipelines-PR-6437-2.
There was a problem hiding this comment.
I'm building your change at jenkins-marathon-pipelines-PR-6437-3.
There was a problem hiding this comment.
✔ Build of #6437 completed successfully.
See details at jenkins-marathon-pipelines-PR-6437-2.
You can create a DC/OS with your patched Marathon by creating a new pull
request with the following changes in buildinfo.json:
"url": "https://s3.amazonaws.com/downloads.mesosphere.io/marathon/builds/1.7.54-544df5d5f/marathon-1.7.54-544df5d5f.tgz",
"sha1": "4a54cd98c72de4ace23a7f1a08a20a8c927bfb4a"You can run system integration test changes of this PR against Marathon
master by triggering this Jenkins job with the Pull_Request_id 6437.
The job then reports back to this PR.
＼\ ٩( ᐛ )و /／
There was a problem hiding this comment.
✔ Build of #6437 completed successfully.
See details at jenkins-marathon-pipelines-PR-6437-3.
You can create a DC/OS with your patched Marathon by creating a new pull
request with the following changes in buildinfo.json:
"url": "https://s3.amazonaws.com/downloads.mesosphere.io/marathon/builds/1.7.55-a3974c5e9/marathon-1.7.55-a3974c5e9.tgz",
"sha1": "d4d0245b41119bf986595516edcf452c3f4f165f"You can run system integration test changes of this PR against Marathon
master by triggering this Jenkins job with the Pull_Request_id 6437.
The job then reports back to this PR.
＼\ ٩( ᐛ )و /／
Summary:
So far Jenkins kills the system test job after two hours. This also
kills our cleanup call. This patch introduces a bash timeout that still
leaves time for a cluster cleanup.
JIRA issues: MARATHON-8364