[AIRFLOW-1889] Move sensors to separate package

[Fokko]

Hi all,

I'd like to move the core sensors to a separate package. Right now all the sensors are contained in
airflow/operators/sensors.py. But I'd like to split this like in /contrib/sensors/:

This also includes splitting the tests which are now mostly in tests/core.py which is a huge test class which is candidate for refactoring.

Make sure you have checked all steps below.

JIRA

• My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
  • https://issues.apache.org/jira/browse/AIRFLOW-XXX

Description

• Here are some details about my PR, including screenshots of any UI changes:

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

• My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

• Passes git diff upstream/master -u -- "*.py" | flake8 --diff

[Fokko]

@bolkedebruin what are your thoughts on this?

Please note that we have two hdfs_sensor.py, both in core and contrib :-)

[Fokko]

Many changes are related to Flake compliancy, I had to shorten a lot of url's and fix the order of imports.

[codecov-io]

Codecov Report

Merging #2875 into master will decrease coverage by 0.76%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2875      +/-   ##
==========================================
- Coverage   73.47%   72.71%   -0.77%     
==========================================
  Files         160      174      +14     
  Lines       12208    12764     +556     
==========================================
+ Hits         8970     9281     +311     
- Misses       3238     3483     +245

Impacted Files	Coverage Δ
airflow/operators/hive_operator.py	70.73% <0%> (-2%)	⬇️
airflow/utils/log/file_processor_handler.py	89.85% <0%> (-1.45%)	⬇️
airflow/www/views.py	71.21% <0%> (-0.51%)	⬇️
airflow/operators/dagrun_operator.py	96.66% <0%> (-0.21%)	⬇️
airflow/operators/sensors.py	67.7% <0%> (-0.12%)	⬇️
airflow/utils/email.py	97.4% <0%> (ø)	⬆️
airflow/task/task_runner/base_task_runner.py	70.31% <0%> (ø)	⬆️
airflow/hooks/hive_hooks.py	40.58% <0%> (ø)	⬆️
airflow/example_dags/example_http_operator.py	100% <0%> (ø)	⬆️
airflow/sensors/sql_sensor.py	93.75% <0%> (ø)
... and 19 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1abe7f6...11550a7. Read the comment docs.

[bolkedebruin]

Can you rebase?

[Fokko]

@bolkedebruin done!

[Fokko]

Rebased

[ashb]

I think sensors.py should still exist, contain a deprecation warning, but still have the sensors available in that package (until 2.0), otherwise DAGs will break with no warning, and it’s easy to maintain back compat in this case

[yati-sagade]

LGTM, this change will make maintenance/testing of sensors much easier. I would love for this to be merged before I hack on AIRFLOW-2001.

[andyxhadji]

Looks great, awesome work! I think we should keep sensors.py for now with a deprecation warning - I can add their removal to the scope of AIRFLOW-1922 for 2.0.

[Acehaidrey]

I'm really glad you guys did this!

[Fokko]

Follow up fix: #2961