Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add --check-container-placement-strategy. #8494

Merged
merged 4 commits into from Nov 25, 2022

Conversation

evanchaoli
Copy link
Contributor

Changes proposed by this PR

We experienced an outage last week. One result of the outage is that all workers over extremely overloaded. Each worker got thousands of containers and volumes, while we have been using "max-active-container" and "max-active-volumes" in container placement strategies.

Then I noticed that, Concourse hard-code "random" strategy for Lidar check containers. While Lidar check containers are recreated every 1 hour, so that in round of recreate, it will create 20K containers.

As a solution, I am adding --check-container-placement-stragegy. We will use max-active-container+max-active-volume+random for check container placement strategy.

  • done

Notes to reviewer

Release Note

@evanchaoli evanchaoli requested a review from a team as a code owner July 25, 2022 12:39
@evanchaoli evanchaoli added this to the v7.9.0 milestone Jul 25, 2022
Signed-off-by: Evan <chaol@vmware.com>
Signed-off-by: Evan <chaol@vmware.com>
Signed-off-by: Evan <chaol@vmware.com>
atc/worker/placement.go Outdated Show resolved Hide resolved
@xtremerui
Copy link
Contributor

we will need PR to update concourse bosh release and helm chart for this new flag.

Signed-off-by: Evan <chaol@vmware.com>
@evanchaoli
Copy link
Contributor Author

we will need PR to update concourse bosh release and helm chart for this new flag.

Copy link
Contributor

@xtremerui xtremerui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@xtremerui xtremerui merged commit 2650921 into concourse:master Nov 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants