Optionally raise an error if source file does not exist in GCSToGCSOperator

[davidpr91]

Description

Right now when using GCSToGCSOperator to copy a file from one bucket to another, if the source file does not exist, nothing happens and the task is considered successful. This could be good for some use cases, for example, when you want to copy all the files from a directory or that match a specific pattern.
But for some other cases, like when you only want to copy one specific blob, it might be useful to raise an exception if the source file can't be found. Otherwise, the task would be failing silently.
My proposal is to add a new flag to GCSToGCSOperator to enable this feature. By default, for backward compatibility, the behavior would be the current one. But it would be possible to force the source file to be required and mark the task as failed if it doesn't exist.

Use case/motivation

Task would fail if the source file to copy does not exist, but only in the case you enable it.

Related issues

If you want to be sure that the source file exists and it will be copied on every execution, currently the operator does not allow you to make the task fail. If the status is successful but nothing is written in the destination, it would be failing silently.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

Assigned you !

[kazanzhy]

Sounds interesting. It's pretty similar to pandas functions parameter - errors={'ignore', 'raise', 'coerce'}. But I see that there are a lot of different operators and sensors in Airflow that help you build complex pipelines with very simple blocks.

I also implemented similar logic to RDS Operators with optional errors in the first iteration but after I decided to keep Operators as straightforward and simple as possible without any logic under the hood.
In this case, GCSObjectExistenceSensor might be useful. If something goes wrong you see it in the UI.