[AIRFLOW-3857] spark_submit_hook cannot kill driver pod in kubernetes

[ogirardot]

Jira

• My PR addresses the following AIRFLOW-3857 issue.

Description

• The spark_submit_hook.on_kill method cannot kill a kubernetes driver pod because of an issue in the client used. The client used in 1.10.{0,1,2} is in fact the CoreV1Api instead of the kubernetes client which has the proper V1DeleteOptions value needed.

Tests

• No test was done on this part, I added a test, but it's not perfect it only mocks the kubernetes client and check the parameters which definitely fails with the last code so... in terms of TDD it works, but it's only a mock call parameter check.

Commits

• My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• No new feature, just a bug fix

Code Quality

• Passes flake8

[ogirardot]

@tswast @kaxil @Fokko another small fix regarding interactions between Airflow and GKE/Kubernetes, any review appreciated

[codecov-io]

Codecov Report

Merging #4678 into master will increase coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #4678      +/-   ##
==========================================
+ Coverage   74.34%   74.36%   +0.01%     
==========================================
  Files         429      429              
  Lines       27950    27951       +1     
==========================================
+ Hits        20779    20785       +6     
+ Misses       7171     7166       -5

Impacted Files	Coverage Δ
airflow/contrib/hooks/spark_submit_hook.py	82.27% <100%> (+2.19%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d11147a...1a90300. Read the comment docs.

[ogirardot]

@tswast @kaxil if you have the time :) any review appreciated

[Fokko]

LGTM, thanks @ogirardot

C02VF05JHV2T:incubator-airflow fdriesprong$ python3
Python 3.7.2 (default, Dec 27 2018, 07:35:06) 
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import kubernetes
>>> kubernetes.client.V1DeleteOptions()
{'api_version': None,
 'dry_run': None,
 'grace_period_seconds': None,
 'kind': None,
 'orphan_dependents': None,
 'preconditions': None,
 'propagation_policy': None}
>>> from airflow.contrib.kubernetes import kube_client
>>> kube_client.V1DeleteOptions()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'airflow.contrib.kubernetes.kube_client' has no attribute 'V1DeleteOptions'