Handle potential race condition in which Trac is notified but git fetch doesn't get the changeset

[rjollos]

Discussed near the end of #8, I continue to see cases in which a changeset does not sync from GitHub and therefore the post-commit hook is not triggered. Running a git fetch and trac-admin $env changeset added for the changeset always triggers the update of the relevant ticket.

The following is always seen in the logs when the issue occurs:

2016-07-12 13:07:06,356 Trac[github] ERROR: Payload contains unknown commit 55cd9488ccd897bb9b75450852c100d13cf0df02

I've considered that it might work to implement a simple retry of git remote update --prune.

[aaugustin]

I think I tried that before without success, but perhaps you'll be more persistent and figure it out :-)

[rjollos]

I see now that the behavior is documented in known issues.

[rjollos]

6f84cdc has some changes that I'll be testing.

[aaugustin]

The best practice for doing this is exponential backoff.

Start with a given delay (order of magnitude: 0,5 - 1s) and at each retry, multiple the delay by a constant factor (order of magnitude: 1,5 - 2). This is easy to implement and nicer on the remote end if they're struggling.

[neverpanic]

I'm not sure the re-try is a very good idea; GitHub seems to have pretty strict timeout policy on webhooks (they regularly mark some of ours as "timeout", even though in quite a few of the cases, Trac seems to have processed them).

Maybe we should add a database table, store the webhook and trigger some asynchronous, delayed processing that can then be re-tried until it succeeds?

[rjollos]

Thanks for the tips. I haven't seen the issue reoccur since this last report. Running my branch with proposed changes, I also never saw log message indicating that the update had to be retried. I'll revisit the issue if it occurs again.

[stuarta]

I'm seeing this approximately every 2 weeks or so with trac 1.2.2.

Just resending the webhook from github doesn't resolve the issue as I still get the "unknown commit"
message in the logs. To resolve I have to manually run git remote update --prune before running a trac-admin <env> repository sync <repo>, after this I can then resend the webhook from github and
trac will get updated.

I will keep trying to understand why we get into this situation.

As per previous comments, I also regularly get timeouts from the github webhooks. Github thinks
they have timed out, but trac does normally get updated. I'm wondering if this is a contributing factor?

I believe we should be accepting the payload from github for async processing (and returning http 202 accepted), then process what github have sent us.