bpf: Double timeout on building BPF programs

[raybejjani]

When a system is slow, or otherwise overloaded, the builds may take
longer. There isn't a fixed number however, so we are trying to be
forgiving.

Fixes #1653

I kept this simple, but it might be useful to have a backoff that goes up with timeouts, so we "discover" the optimum balance.

[joestringer]

If we think that a more interesting solution to this should be done at some point, would you consider filing an issue for that?

[raybejjani]

@joestringer Do you mean file an issue now explaining the problem, and when a more complex solution would be needed? or wait and see then decide to make the ticket?
In any case, my expectation is that we will discover how often this occurs once we begin collecting metrics and when we see more deployments. I'm supposed to add metrics for all the things in the next week :)
Did you have something else in mind? I'm happy to make an issue out of it now if you want.

[joestringer]

The "it might be useful to..." sentence in the description sounded like a future task that may or may not be tackled, depending on priorities, etc. If you're already planning to do the metrics stuff then it shouldn't be necessary.

[raybejjani]

Ah, I see. I had a "double the backoff on each failure" scheme in mind, but I would prefer to have some metrics around the build time to begin with. In fact, thinking on it more, if we collect metrics anyway (a task I have on my list already) then we can make the backoff 90th percentile or something like it to ensure most builds succeed but we don't wait forever for the outliers.