Issue with system test - tests/system/providers/microsoft/azure/example_azure_blob_to_gcs.py

[Adaverse]

Apache Airflow version

2.6.1

What happened

I was going through tests/system/providers/microsoft/azure/example_azure_blob_to_gcs.py, and after grasping the motive behind the system test, one thing is that each of these tests should be self-sufficient dags and should act as a good example for that specific object (sensor, operator, etc...) under test. In the above case, the dag flow -

    (
        wait_for_blob
        >> wait_for_blob_async
        >> wait_for_blob_prefix
        >> wait_for_blob_prefix_async
        >> transfer_files_to_gcs
    )

Please correct me if I'm wrong, but isn't this test waiting for a blob to exist before testing the main operator (that it is supposed to test) i.e. AzureBlobStorageToGCSOperator. Hence waiting for some external service (or some dag or task) to create the blob to continue with the test. Or am I missing something here or is the part of the pre-configuration that is needed to be done for this - just not documented?

According to me, it should have been something like this -

        create_blob_azure
        >> transfer_files_to_gcs
        >> check_in_gcs    # for successful transfer

What you think should happen instead

No response

How to reproduce

.

Operating System

.

Versions of Apache Airflow Providers

.

Deployment

Other

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Adaverse]

cc- @phanikumv

[phanikumv]

I agree with you that this example DAG should just showcase how to move an existing blob to GCS. So please go ahead and create PR for this. I don't think you need to create issues for these things, unless it is a bug or a new feature :-)