Trigger Resources: Per-job triggers

[matthewpereira]

TLDR

Trigger resources allow jobs to specify parameters that can trigger new builds but don't have anything to fetch - they just propagate config fragments to the build.

Problem

• Currently, Concourse only allows for one time interval configured in a pipeline, as opposed to letting everything fire on its own interval. This can cause thundering herd problems, especially for those utilizing Concourse at large scale.

Proposed Solution

• Introduce two new step types - load_var and get_varalong with a new mechanism for builds to use a 'local var source' at runtime.
• The load_var step can be used top read a value from a file at runtime and use it as a local var in subsequent steps
• The get_var step can be used to fetch a var from a var source and trigger a new build when its value changes

Outcomes

• Improve today's usage of the time resource by having per-job trigger semantics rather than having all jobs downstream of one time resource leading to a thundering herd of builds hitting workers all at once.
• Resource (proto)type authors will no longer have to implement *_file forms of any of their parameters
• The get_var step introduces more flexible way to trigger and parameterize jobs without having to use a resource
• Allow the team to un-feature-flag Equivalent resources defined across pipelines and teams should only correspond to a single version history #2386 , tying equivalent resources defined across pipelines and teams to a single version history. This would reduce wasted space in the DB and redundant resource checking, which is an increasingly large win for teams of increasingly large scale.

Current Status

• Engineering implementation to be prioritized in Q3

References

See RFC
See RFC Discussion