Skip to content

Validate weather-dl target location before starting the job #209

@blackvvine

Description

@blackvvine

When the weather-dl job is submitted with an invalid path, e.g. a local path instead of GCS when using the Dataflow runner, the job never throws an error. Instead, it gets stuck in the exponential backoff loop of the Fetcher.upload() method.

Symptoms

GCS write exponential backoff WARN message:

Retry with exponential backoff: waiting for 295.9412769663183 seconds before retrying open because we caught exception: FileNotFoundError: [Errno 2] No such file or directory: ''

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions