Add support for Microsoft Azure Blob Storage in Google Cloud Storage Transfer Service operators

[mik-laj]

Description

Hello,

We should add support for Microsoft Azure Blob Storage in CloudDataTransferServiceCreateJobOperator. This support should allow pass credentials to Azure from the connection. A similar feature already exists for AWS.
https://github.com/apache/airflow/blob/42eef38/airflow/providers/google/cloud/operators/cloud_storage_transfer_service.py#L48-L57
This issue is due to the March 23 service update.

To complete this task you must complete the following steps:

• Update the operator
• Update the example DAG
• Update the documentation
• Run system tests (final check)

If you haven't used the GCP yet, after creating the account you will get $300, which will allow you to get to know these services better.

The implementation of this task will allow a better understanding of GCP services, as well as learn methods of testing that is required by the community. If anyone is interested in this task, I am willing to provide all the necessary tips and information.

Use case / motivation

N/A

Related Issues

N/A

[khyurri]

Please assign to me

[khyurri]

def __init__(
        self,
        body: dict,
        aws_conn_id: str = 'aws_default',
        gcp_conn_id: str = 'google_cloud_default',
        api_version: str = 'v1',
        *args,
        **kwargs
    )

In such a method, it’s rather difficult to nicely add support for something other than aws_conn_id, while maintaining backward compatibility.

I see 2 options

1. Add the msa_conn_id parameter befor *args. In this case, if msa_conn_id is passed, use it, otherwise aws_conn_id. msa_conn_id cannot be set to the default value.

2. Divide CloudDataTransferServiceCreateJobOperator into 2 operators: for AWS and for Azure. This leave for compatibility for AWS, may declare it obsolete

Which option is more acceptable?

[mik-laj]

All parameters to all operators must be called using keywords arguments. Please look at: airflow.utils.decorators.apply_defaults

It should be something similar to:

    @apply_defaults
    def __init__(
        self,
        body: duct,
        azure_conn_id: str = 'azure_default',
        aws_conn_id: str = 'aws_default',
        gcp_conn_id: str = 'google_cloud_default',
        api_version: str = 'v1',
        *args,
        **kwargs
    ) -> None:

[khyurri]

In this case:

azure_conn_id: str = 'azure_default',
aws_conn_id: str = 'aws_default',

which connection should be used? Both?

[mik-laj]

It depends on the value of the body parameter.
Please look at this code related to AWS
https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/operators/cloud_storage_transfer_service.py#L48-L57
Here is example GCS => GCS transfer body:
https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/example_dags/example_cloud_storage_transfer_service_gcp.py#L60-L74
Here is example AWS => GCS transfer body:
https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/example_dags/example_cloud_storage_transfer_service_aws.py#L81

[khyurri]

I have not found default connection for azure_blob_storage.
I'll add it into create_default_connections(session=None) in airflow/utils/db.py. As I see, this function is called only from initdb.

Is there any possibility for users to update default connections during the airflow upgrade?

[mik-laj]

Can you use an existing connection for Azure Blob Storage? https://airflow.readthedocs.io/en/latest/_api/airflow/providers/microsoft/azure/operators/wasb_delete_blob/index.html#module-airflow.providers.microsoft.azure.operators.wasb_delete_blob
This operators uses wasb_default. In the previous comment I used azure_default only as an example. In the future, we may be able to combine multiple connection names for different Azure services, but I'm not an Azure expert.

Why do you need to add a new connection?

[khyurri]

Yeah, just realized, that Blob Store called wasb. Sure, I'll use this connection.

[khyurri]

Still in progress. I need to write tests and documentation :) No estimates yet.

[mik-laj]

@joppevos How's the documentation writing going? Do you have any problem? I will gladly help.

[khyurri]

@mik-laj I will complete issue at nearest holidays. Thanx )

[FHoffmannCode]

Hi, @khyurri are you still working on this issue?

[khyurri]

Hi, @FHoffmannCode
It's completed. Review is needed.

[turbaszek]

@khyurri can you please share a link to PR?

[khyurri]

@turbaszek
Sure, #8818

[eladkal]

@khyurri unassigning you as you did not complete the PR.