Implement retry when failing to write to GCS

[foolip]

Check

A link to the GitHub Check run containing my issue:
https://github.com/web-platform-tests/wpt/pull/15475/checks

Description

There are checks for staging.wpt.fyi, but none for wpt.fyi.

[foolip]

The runs were ingested: https://wpt.fyi/results/?sha=ed55be4

[lukebjerring]

In AppEngine > Services > default > Logs

API error: HTTP 500: error(s) occured when transferring results from azure to GCS:
Post https://www.googleapis.com/upload/storage/v1/b/wptd-results-buffer/o?alt=json&projection=full&uploadType=multipart: API error 1 (urlfetch: INVALID_URL)

FYI @Hexcles . Not sure if is this a transient error or not - more inspection required.

[Hexcles]

It is. I've seen this error occasionally. We don't directly post to GCS using urlfetch; this is done by the appengine GCS library internally which we have no control over. We might need some retry mechanism, but I'm not sure yet which part we need to retry (Write, Close, or both).

[foolip]

I have no results on web-platform-tests/wpt#15577 either. Possibly they're still in the pipeline, we'll see. What's the expected delay?