Refactor Code to Better control Beam Retries

[damccorm]

Uploads might be retried in an underlying library on an upload thread.

beam/sdks/python/apache_beam/io/gcp/gcsio.py

Lines 635 to 639 in c026c9f

	# TODO(silviuc): Refactor so that retry logic can be applied.
	# There is retry logic in the underlying transfer library but we should make
	# it more explicit so we can control the retry parameters.
	@retry.no_retries # Using no_retries marks this as an integration point.
	def _start_upload(self):

 

The code is written in such a way as to not easily allow retries. One way to do so would be to allow http errors to be raised and then retry.

 

See also https://issues.apache.org/jira/browse/BEAM-12567

Imported from Jira BEAM-12758. Original Jira may contain additional context.
Reported by: Ryan.Thompson.