[AIRFLOW-839] Only log status for DockerOperator pull if the key is present

[iamthebot]

Jira

• My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
  • https://issues.apache.org/jira/browse/AIRFLOW-XXX
  • In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.

Description

This addresses AIRFLOW-839 wherein a job will fail on logging the status of a docker image pull and fail to provide further output.

Example:

{docker_operator.py:160} INFO - Downloading
[2018-10-25 02:10:44,746] {base_task_runner.py:93} INFO - Subtask: [2018-10-25 02:10:44,746] {docker_operator.py:160} INFO - Verifying Checksum
[2018-10-25 02:10:44,746] {base_task_runner.py:93} INFO - Subtask: [2018-10-25 02:10:44,746] {docker_operator.py:160} INFO - Download complete
[2018-10-25 02:10:45,034] {base_task_runner.py:93} INFO - Subtask: [2018-10-25 02:10:45,033] {models.py:1544} ERROR - 'status'
[2018-10-25 02:10:45,034] {base_task_runner.py:93} INFO - Subtask: Traceback (most recent call last):
[2018-10-25 02:10:45,035] {base_task_runner.py:93} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1457, in _run_raw_task
[2018-10-25 02:10:45,035] {base_task_runner.py:93} INFO - Subtask:     result = task_copy.execute(context=context)
[2018-10-25 02:10:45,035] {base_task_runner.py:93} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/operators/docker_operator.py", line 160, in execute
[2018-10-25 02:10:45,035] {base_task_runner.py:93} INFO - Subtask:     logging.info("{}".format(output['status']))
[2018-10-25 02:10:45,035] {base_task_runner.py:93} INFO - Subtask: KeyError: 'status'
[2018-10-25 02:10:45,037] {base_task_runner.py:93} INFO - Subtask: [2018-10-25 02:10:45,037] {models.py:1571} INFO - All retries failed; marking task as FAILED
[2018-10-25 02:10:45,170] {base_task_runner.py:93} INFO - Subtask: [2018-10-25 02:10:45,170] {email.py:110} INFO - Sent an alert email to <redacted>
[2018-10-25 02:10:45,316] {base_task_runner.py:93} INFO - Subtask: [2018-10-25 02:10:45,316] {models.py:1594} ERROR - 'status'
[2018-10-25 02:10:45,316] {base_task_runner.py:93} INFO - Subtask: Traceback (most recent call last):
[2018-10-25 02:10:45,317] {base_task_runner.py:93} INFO - Subtask:   File "/usr/local/bin/airflow", line 28, in <module>
[2018-10-25 02:10:45,317] {base_task_runner.py:93} INFO - Subtask:     args.func(args)
[2018-10-25 02:10:45,317] {base_task_runner.py:93} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 438, in run
[2018-10-25 02:10:45,317] {base_task_runner.py:93} INFO - Subtask:     pool=args.pool,
[2018-10-25 02:10:45,317] {base_task_runner.py:93} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/utils/db.py", line 49, in wrapper
[2018-10-25 02:10:45,317] {base_task_runner.py:93} INFO - Subtask:     result = func(*args, **kwargs)
[2018-10-25 02:10:45,317] {base_task_runner.py:93} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1457, in _run_raw_task
[2018-10-25 02:10:45,317] {base_task_runner.py:93} INFO - Subtask:     result = task_copy.execute(context=context)
[2018-10-25 02:10:45,318] {base_task_runner.py:93} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/operators/docker_operator.py", line 160, in execute
[2018-10-25 02:10:45,318] {base_task_runner.py:93} INFO - Subtask:     logging.info("{}".format(output['status']))
[2018-10-25 02:10:45,318] {base_task_runner.py:93} INFO - Subtask: KeyError: 'status'
[2018-10-25 02:11:08,673] {jobs.py:183} DEBUG - [heartbeat]
[2018-10-25 02:11:08,673] {jobs.py:2436} INFO - Task exited with return code 1

We already log the rest of the status of the pull so this wouldn't introduce a regression.

• Here are some details about my PR, including screenshots of any UI changes:

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:
  This is just checking for the key presence prior to logging. It would be difficult to unit test this outside of a basic dictionary check which is rather meaningless.

Commits

• My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.

Code Quality

• Passes flake8

[codecov-io]

Codecov Report

Merging #4095 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #4095      +/-   ##
==========================================
+ Coverage   77.91%   77.91%   +<.01%     
==========================================
  Files         199      199              
  Lines       15956    15957       +1     
==========================================
+ Hits        12432    12433       +1     
  Misses       3524     3524

Impacted Files	Coverage Δ
airflow/operators/docker_operator.py	97.64% <100%> (+0.02%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 19837bc...b954c0d. Read the comment docs.

[KevinYang21]

LGTM

[nikhilsimha]

Looks great! Thanks @iamthebot