Added Plexus as an Airflow provider

[millertracy]

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, pylint and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://apache-airflow-slack.herokuapp.com/

[potiuk]

Hey @millertracy -> reviewed it. Nice piece of provider :).

The main comment is about the settup.py change (plexus needs 'plexus' extra and arrow should be added there - also then plexus extra should be added to "all_devs/all" extras similarly to other extras.

Also then you will need to mark it as "provider" dependency in PROVIDERS_REQUIREMENTS.

This should likely fix backport package problem that you have (airflow 1.10.10 does not have arrow as requirement, neither the generated backport package).

There is also some work to make the docs work - some spellcheck problems from what I see.

One more NIT - It would also be great if you create a HowTo out of your example. You can see how other operators do that, but basically it is a very short "Howto" doc that extracts part of the "example_dag" and adds some description on what the operator does.

Another part (but this is something that we have no full CI automation yet) is to turn the example dag into a runnable "system test". This is described in https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-system-tests - and it's not required, but it seems to be easy to do in your case as this is rather simple example you have.

We plan to automate system tests later this year and it would be great to have such easy-to-run system test in place. But this one is nice-to-havee.

This is actually the most powerful part of the exmple_dags

[millertracy]

Hey @millertracy -> reviewed it. Nice piece of provider :).

The main comment is about the settup.py change (plexus needs 'plexus' extra and arrow should be added there - also then plexus extra should be added to "all_devs/all" extras similarly to other extras.

Also then you will need to mark it as "provider" dependency in PROVIDERS_REQUIREMENTS.

This should likely fix backport package problem that you have (airflow 1.10.10 does not have arrow as requirement, neither the generated backport package).

There is also some work to make the docs work - some spellcheck problems from what I see.

One more NIT - It would also be great if you create a HowTo out of your example. You can see how other operators do that, but basically it is a very short "Howto" doc that extracts part of the "example_dag" and adds some description on what the operator does.

Another part (but this is something that we have no full CI automation yet) is to turn the example dag into a runnable "system test". This is described in https://github.com/apache/airflow/blob/master/TESTING.rst#airflow-system-tests - and it's not required, but it seems to be easy to do in your case as this is rather simple example you have.

We plan to automate system tests later this year and it would be great to have such easy-to-run system test in place. But this one is nice-to-havee.

This is actually the most powerful part of the exmple_dags

@potiuk thank you! I will make those changes. will i also need to add backward compatibility for airflow 1.10* by adding plexus hook/operator to the airflow.contrib package?

[potiuk]

@potiuk thank you! I will make those changes. will i also need to add backward compatibility for airflow 1.10* by adding plexus hook/operator to the airflow.contrib package?

No need. By adding it to the setup properly, backport provider is automatically created and tested, so no need to do anything.

[millertracy]

@potiuk thank you! I will make those changes. will i also need to add backward compatibility for airflow 1.10* by adding plexus hook/operator to the airflow.contrib package?

No need. By adding it to the setup properly, backport provider is automatically created and tested, so no need to do anything.

@potiuk That's great. Double checking though - i will need to reference the 1.10* compatibility in the README.md like the other providers correct? By referencing "Moved Hooks and Operators"?

[millertracy]

@potiuk finished changes. is there anything else that needs to be done?

[potiuk]

Almost perfect! Just remove the README file. We are going to generate one automatically when we release next wave of backports.

[millertracy]

@potiuk thanks for approving changes. 2 checks were cancelled, but it doesn't look related to my code. When should i expect a merge into the base branch?

[potiuk]

Cool stuff @millertracy ! Thanks for that! "Have One more provider we" as the old Yoda would say.

[boring-cyborg]

Awesome work, congrats on your first merged pull request!