Intelligent resource checking based on the new version history

[ukabu]

What challenge are you facing?

We have more than 160K resource-checks/hour in out largest Concourse instance.

They take a significant amount of computing resource to perform. Most of those resource checks find no new version. So it just waste computing resource.

What would make this better?

To have something like check_every: predict. That would starts with the default polling interval and learn, for each resource, how frequently and when new version arrives. It will then adjust the polling frequency so it polls more frequently when new version are more likely to appear and less frequently when it's less likely.

Are you interested in implementing this yourself?

Very, but lack Go knowledge and time.

[marco-m]

Did you consider using webhooks?

[ukabu]

webhooks are ok for git repo, but not applicable for many kind of resources. also, they are not really applicable for resources that are used in many pipelines (1 to many triggers).

[clarafu]

@ukabu Are you on Concourse v5.x? If so, have you tried enabling global resources? It will mostly only help if you have a lot of duplicated resources throughout your teams though.

[ukabu]

yes we are on 5.2 as of today (been on 5.1 for a week). We haven't enabled it yet, the upgrade from 4.2.3 was painful and we will enable additional feature like global resource, vault cache and other one by one and make sure that everything is stable.
Global resource will certainly help a bit, but I don't think this will reduce resource checks dramatically.

[kcmannem]

@ukabu just curious how many workers do you have and what size?

[chendrix]

From my uninformed outside opinion it feels like this behavior might actually be preferable as the default, with the option to override to known intervals.

[vito]

@chendrix Yeah it's interesting to think about. It could certainly be more sensible than the current arbitrary default of '1 minute'.

I wonder what the change frequency might look like for real-world use cases though. For example, git repos may have pretty irregular change frequencies, and there's also the fact that everything probably slows to a halt over the weekend when people aren't working. We could probably find reasonable defaults to handle this behavior though, e.g. have an upper bound and just deal with the initial slow checking on Monday.

In any case it's probably a reasonable thing to want from an operator's perspective, and if users want something more immediate we could take another swing at making the webhooks UX less awful so people will use it more.

[ukabu]

we have 40 workers on Azure with 4 CPU and 32G of memory. +1 tagged worker with larger disks for of bosh backup tasks

[chendrix]

Wouldn't some form of really slow, multi-sample exponential backoff be sufficient? And when a check container finds a hit, you reset the exponential backoff?

[ukabu]

@vito every night and on weekend we have a very low commit frequency. But it also vary by repo. We have some apps repo that may be very active now, but will have less activity a month from now as they become more stable and have less features being added.
Just automatically reducing the rate of resource checks to a max of 10 times the default based on new version frequency would probably reduce dramatically the number of resource check per hour.
This is an very interesting machine learning problem :-D

[ukabu]

@chendrix that may work.

[stale]

Beep boop! This issue has been idle for long enough that it's time to check in and see if it's still important.

If it is, what is blocking it? Would anyone be interested in submitting a PR or continuing the discussion to help move things forward?

If no activity is observed within the next week, this issue will be exterminated closed, in accordance with our stale issue process.