updatebot making tons of commits to an existing PR in rapid succession causes pipelinerunner to fall over

[abayer]

{"level":"fatal","msg":"failed to clone three times it's likely things wont recover so lets kill the process after 3 attempts, last error: failed to fetch [pull/5766/head:PR-5766 0ffa7b5c253153cb70d9c41aa95c8b76766d4ceb] from https://github.com/jenkins-x/jenkins-x-platform.git in directory /tmp/git204215515: git output: error: Server does not allow request for unadvertised object 0ffa7b5c253153cb70d9c41aa95c8b76766d4ceb: failed to run 'git fetch origin --depth=1 pull/5766/head:PR-5766 0ffa7b5c253153cb70d9c41aa95c8b76766d4ceb' command in directory '/tmp/git204215515', output: 'error: Server does not allow request for unadvertised object 0ffa7b5c253153cb70d9c41aa95c8b76766d4ceb'","time":"2019-07-17T18:53:46Z"}

Something about the rapid-fire succession of commits causes problems with the checkout. Ideally we'd either batch them or not open the PR until we've made all the commits.

[cagiti]

{"level":"fatal","msg":"failed to clone three times it's likely things wont recover so lets kill the process after 3 attempts, last error: failed to fetch [pull/5766/head:PR-5766 0ffa7b5c253153cb70d9c41aa95c8b76766d4ceb] from https://github.com/jenkins-x/jenkins-x-platform.git in directory /tmp/git204215515: git output: error: Server does not allow request for unadvertised object 0ffa7b5c253153cb70d9c41aa95c8b76766d4ceb: failed to run 'git fetch origin --depth=1 pull/5766/head:PR-5766 0ffa7b5c253153cb70d9c41aa95c8b76766d4ceb' command in directory '/tmp/git204215515', output: 'error: Server does not allow request for unadvertised object 0ffa7b5c253153cb70d9c41aa95c8b76766d4ceb'","time":"2019-07-17T18:53:46Z"}

Something about the rapid-fire succession of commits causes problems with the checkout. Ideally we'd either batch them or not open the PR until we've made all the commits.

I think the latter approach makes sense 👍

[abayer]

cc @pmuir

[abayer]

This combined with pipelinerunner crashing enough times in a row sends the pod into crash back off, meaning nothing can build at all for a while. Ow.

[pmuir]

Simplest is to improve the updatebot command so it can make one commit where it's currently making lots - that's just a limitation in the way it does updates.

It would be very hard to make it batch them somehow as they are not all connected processes. That would probably be easier to address by introducing some sort of delay/sleep/quiet period in the controller.

[pmuir]

There is also no reason to kill the pipeline runner in this scenario. We are killing it to handle those odd errors we have with git where there is no output IIRC.

[abayer]

We're killing pipelinerunner 'cos we're still on such an old pipelinerunner that it doesn't have the fix from like two months ago to ignore the unadvertised object error.

[abayer]

(which reminds me that we should probably also ignore merge conflicts there - next time I see one show up in the logs so that I can get the error message right, I'll open a PR for that)

[abayer]

2aa2225 is the relevant commit that made pipelinerunner stop panicking for unadvertised object, fwiw.

[abayer]

oh, wait, we don't panic over merge conflicts. Ignore me on that front. =)

[abayer]

I'm gonna close this - we solved the immediate problem, which was more usage than mechanical.