Add support for Microsoft Azure Blob Storage in Google Cloud Storage Transfer Service operators

[khyurri]

Operators Code, examples and documentation completed. I've got bug on CloudDataTransferServiceCancelOperationOperator from example. I'll fix it.

Can you, please, review changed code?

Make sure to mark the boxes below before creating PR: [x]

• Description above provides context of the change
• Unit tests coverage for changes (not needed for documentation changes)
• Target Github ISSUE in description if exists
• Commits follow "How to write a good git commit message"
• Relevant documentation is updated including usage instructions.
• I will engage committers as explained in Contribution Workflow Example.

---

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

[mik-laj]

@khyurri Do you need any help? I am available for you if you have any questions.

[khyurri]

@mik-laj
yes, I think I need help with previewing documentation. I cannot build documentation locally and cannot check it is correct

[ad-m]

@khyurri , how do you attempt to build documentation and what errors are you getting?

[potiuk]

Indeed we have a super-easy way to build documentation - simply run './breeze build-documentation' and all should happen automatically. I would love to hear what problems you experience with it.

[khyurri]

@ad-m
I build documentation using this command from repository root :

python setup.py build_sphinx

Build completes without any errors with message: The HTML pages are in build/sphinx/html.

I can find separate documentation pages (like in operators/cloud_storage_transfer_service/index.html), but cannot find example usages (like howto_operator_gcp_transfer_create_job_body_azure).

[potiuk]

@khyurri - here is the official way of building the documentation (described in our CONTRIBUTING document):
https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#documentation

Additionally (if you do not want to setup your virtualenv) we have BREEZE development environment that downloads and uses Docker image to do the same: https://github.com/apache/airflow/blob/master/BREEZE.rst#using-the-airflow-breeze-environment

[khyurri]

@potiuk @ad-m
I builded documentation locally, but I can't find example html for this file. airflow/providers/google/cloud/example_dags/example_cloud_storage_transfer_service_azure.py

How examples are displayed in the documentation?
As far as I understand this documentation is displayed here https://airflow.apache.org/docs/stable/howto/operator/gcp/transfer.html.
Is it enough for me to check that the examples from the documentation work, or should I do something else?

[mik-laj]

@khyurri Sorry. I missed your messsage. Example DAG files are not available in the documentation until they are used in the documentation. You can attach a fragment of files using the exampleinclude directive. Documentation for this service can be found here.
https://github.com/apache/airflow/blob/master/docs/howto/operator/gcp/cloud_storage_transfer_service.rst

Is it enough for me to check that the examples from the documentation work, or should I do something else?

I would be happy if you run example DAG and see if it works. You can do it with the help of automatic system tests.

I am very sad that you had to wait so long for my answer. I hope that my inattention will discourage you from completing this change.

[khyurri]

I think it's done.

[turbaszek]

@khyurri can you rebase please? In the meantime we introduced black formatter for providers package. Sorry for such a long time of waiting for review.

CC: @michalslowikowski00 may be interested in this PR

[RosterIn]

@khyurri seems like rebase wasn't successful

[khyurri]

seems like rebase wasn't successful

Yeah. Working on it.

[mik-laj]

@khyurri Do you need help? I will gladly do it for you.

[khyurri]

@mik-laj
Hey, sorry for delay. Could you please help me to rebase this PR?

[ad-m]

@khyurri do you have experience in rebase?

/sparkle @mik-laj for supporting Apache Airflow project!

[mik-laj]

Hey, sorry for delay. Could you please help me to rebase this PR?

I was busy with Airflow 2.0, but today I will try to look at it.

[khyurri]

I rebased on master, black changed too much code.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[mik-laj]

Suggested change

	delete_transfer_from_aws_job = CloudDataTransferServiceDeleteJobOperator(
	delete_transfer_from_job = CloudDataTransferServiceDeleteJobOperator(

[mik-laj]

Suggested change

	task_id="delete_transfer_from_aws_job",
	task_id="delete_transfer_from_job",

[mik-laj]

Suggested change

	cancel_operation >> delete_transfer_from_aws_job
	cancel_operation >> delete_transfer_from_job

[mik-laj]

Code looks good to me. Can you add tests to avoid regression?

[khyurri]

Sure

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.