Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add optional skew to interval #22

Closed
wants to merge 1 commit into from

Conversation

phillbaker
Copy link

This adds a skew parameter to be used with intervals whose required time precision is lower, reducing the simultaneous load on workers when multiple tasks are kicked off at the same time.

Example usage:

resources:
- name: 5-ish-min
  type: time
  source:
    interval: 5m
    skew: 1m

jobs:
- name: something-every-4-to-6m
  plan:
  - get: 5-ish-min
    trigger: true
  - task: something
    config: # ...

@vito
Copy link
Member

vito commented Dec 12, 2017

Thanks for the PR!

I don't think this will help much though. When there are 10 jobs all fetching from the same 'skewed' resource (as the skew would only affect the version discovery, not the resulting scheduling), you'll still end up with a stampede.

This may be better addressed at the root of the concern, which is scheduling workloads across jobs.

There's also serial_groups, though that will eliminate all parallelism.

Related: concourse/concourse#675

@vito vito closed this Dec 12, 2017
@phillbaker
Copy link
Author

Thanks for taking a look. We were on an old version of concourse and in the 3.0 refactor to reuse containers this does become useless.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants