Skip to content

Commit

Permalink
Allow 30m more to acquire consul lock (#2922)
Browse files Browse the repository at this point in the history
We seem to be having more frequent consul lock timeouts on CI. Best
guess, this is a bandwidth issue of all PRs using the same e2e
clusters and a higher number of PRs, plus probably some organic growth
in e2e length. This PR is a stop-gap:

* Bumps the consul lock timeout from 1h to 1.5h - basically "wait our
turn" for longer.

* Bumps the whole build timeout from 2h to 2.5h. I assume the whole
build has a generous timeout to allow for the image build, etc, prior
to e2es, so kept the increase here 1:1.

* Bumps the queue TTL from (default) 1h to 6h. This is a generous bump
to allow builds to hang out longer in the queue.
  • Loading branch information
zmerlynn committed Jan 24, 2023
1 parent 03b9799 commit c12c9f2
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 2 deletions.
2 changes: 1 addition & 1 deletion build/e2e-image/entrypoint.sh
Expand Up @@ -36,7 +36,7 @@ kubectl port-forward statefulset/consul-consul-server 8500:8500 &
echo "Waiting consul port-forward to launch on 8500..."
timeout 60 bash -c 'until printf "" 2>>/dev/null >>/dev/tcp/$0/$1; do sleep 1; done' 127.0.0.1 8500
echo "consul port-forward launched. Starting e2e tests..."
echo "consul lock -child-exit-code=true -timeout 1h -verbose LockE2E '/root/e2e.sh "$FEATURES" "$CLOUD_PRODUCT" "$REGISTRY"'"
echo "consul lock -child-exit-code=true -timeout 90m -verbose LockE2E '/root/e2e.sh "$FEATURES" "$CLOUD_PRODUCT" "$REGISTRY"'"
consul lock -child-exit-code=true -timeout 1h -verbose LockE2E '/root/e2e.sh "'$FEATURES'" "'$CLOUD_PRODUCT'" "'$REGISTRY'"'
killall -q kubectl || true
echo "successfully killed kubectl proxy"
3 changes: 2 additions & 1 deletion cloudbuild.yaml
Expand Up @@ -381,7 +381,8 @@ substitutions:
_RUST_SDK_BUILD_CACHE_KEY: rust-sdk-build
_REGISTRY: us-docker.pkg.dev/${PROJECT_ID}/ci
tags: ['ci']
timeout: 7200s
timeout: 9000s # 2.5h
queueTtl: 21600s # 6h
images:
- '${_REGISTRY}/agones-controller'
- '${_REGISTRY}/agones-sdk'
Expand Down

0 comments on commit c12c9f2

Please sign in to comment.