When the weather-dl job is submitted with an invalid path, e.g. a local path instead of GCS when using the Dataflow runner, the job never throws an error. Instead, it gets stuck in the exponential backoff loop of the Fetcher.upload() method.
Symptoms
GCS write exponential backoff WARN message:
Retry with exponential backoff: waiting for 295.9412769663183 seconds before retrying open because we caught exception: FileNotFoundError: [Errno 2] No such file or directory: ''