[AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod

[petedejoy]

---

Issue link: AIRFLOW-6843

If you're using the Kubernetes executor, Amazon EKS deletes pods very quickly after tasks complete. This can be an issue if you're scraping Airflow logs from a service like FluentD, as it means that the task pod gets deleted before FluentD can pick up logs for fast-running (or fast-failing) tasks.

The kube_client_request_args environment variable is passed to the delete_namespaced_pod client request, but we need a way to pass the grace_period_seconds param (and potentially others) to the V1DeleteOptions included in the body of the request. Towards that objective, I've created a delete_option_kwargs object that can be set via an Airflow environment variable. This is object is then passed to the V1DeleteOptions of the delete_namespaced_pods request.

This can be set via an AIRFLOW__KUBERNETES__DELETE_OPTION_KWARGS env var. The value should be JSON and can contain any of the options available in the Kube client V1DeleteOptions class.
Make sure to mark the boxes below before creating PR: [x]

• Description above provides context of the change
• Commit message/PR title starts with [AIRFLOW-NNNN]. AIRFLOW-NNNN = JIRA ID^*
• Unit tests coverage for changes (not needed for documentation changes)
• Commits follow "How to write a good git commit message"
• Relevant documentation is updated including usage instructions.
• I will engage committers as explained in Contribution Workflow Example.

^* For document-only changes commit message can start with [AIRFLOW-XXXX].

---

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

[dimberman]

@petedejoy can you fix the conflicts?

[dimberman]

@petedejoy if you want a unit test for this, you can set the timeout to 15 seconds and make sure that once a pod completes it stays up for 15 seconds. @kaxil do you think a test is necessary here?

[codecov-io]

Codecov Report

❗ No coverage uploaded for pull request base (master@b6a11c1). Click here to learn what that means.
The diff coverage is 93.18%.

@@            Coverage Diff            @@
##             master    #7523   +/-   ##
=========================================
  Coverage          ?   86.85%           
=========================================
  Files             ?      896           
  Lines             ?    42638           
  Branches          ?        0           
=========================================
  Hits              ?    37035           
  Misses            ?     5603           
  Partials          ?        0

Impacted Files	Coverage Δ
airflow/utils/helpers.py	82.6% <100%> (ø)
airflow/settings.py	93.66% <100%> (ø)
airflow/models/dagbag.py	89.61% <100%> (ø)
...s/google/cloud/example_dags/example_stackdriver.py	100% <100%> (ø)
airflow/models/taskinstance.py	95.17% <100%> (ø)
...ow/providers/microsoft/azure/hooks/azure_cosmos.py	76.72% <100%> (ø)
airflow/providers/amazon/aws/hooks/s3.py	96.6% <100%> (ø)
...ow/providers/google/cloud/operators/stackdriver.py	100% <100%> (ø)
airflow/models/dag.py	91.37% <100%> (ø)
airflow/utils/log/logging_mixin.py	95.38% <100%> (ø)
... and 7 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b6a11c1...e13f6c1. Read the comment docs.

[ashb]

Doesn't affect the output, but could be done as:

Suggested change

	example: "{{\"grace_period_seconds\": 10}}"
	example: '{{"grace_period_seconds": 10}}'

Also: OH GOD. Do we show this as { or {{ when we render the docs. Becasuse it needs to be { -- {{ is only needed in default_airflow.cfg because that is a template, but we should show in our docs is not templated/formated.

[ashb]

Maybe we need to update the pre-commit script to replace { with {{ and } with }} so that this and the default_airflow.cfg are correct.

[kaxil]

It is tricky because of task_log_prefix_template = {{ti.dag_id}}-{{ti.task_id}}-{{execution_date}}-{{try_number}}. It would replcae it too task_log_prefix_template = {{{{ti.dag_id}}}}-{{{{ti.task_id}}}}-{{{{execution_date}}}}-{{{{try_number}}}}

Maybe we need to update the pre-commit script to replace { with {{ and } with }} so that this and the default_airflow.cfg are correct.

[kaxil]

Updated !

[codecov-io]

Codecov Report

❗ No coverage uploaded for pull request base (master@b6a11c1). Click here to learn what that means.
The diff coverage is 93.18%.

@@            Coverage Diff            @@
##             master    #7523   +/-   ##
=========================================
  Coverage          ?   86.85%           
=========================================
  Files             ?      896           
  Lines             ?    42638           
  Branches          ?        0           
=========================================
  Hits              ?    37035           
  Misses            ?     5603           
  Partials          ?        0

Impacted Files	Coverage Δ
airflow/utils/helpers.py	82.6% <100%> (ø)
airflow/settings.py	93.66% <100%> (ø)
airflow/models/dagbag.py	89.61% <100%> (ø)
...s/google/cloud/example_dags/example_stackdriver.py	100% <100%> (ø)
airflow/models/taskinstance.py	95.17% <100%> (ø)
...ow/providers/microsoft/azure/hooks/azure_cosmos.py	76.72% <100%> (ø)
airflow/providers/amazon/aws/hooks/s3.py	96.6% <100%> (ø)
...ow/providers/google/cloud/operators/stackdriver.py	100% <100%> (ø)
airflow/models/dag.py	91.37% <100%> (ø)
airflow/utils/log/logging_mixin.py	95.38% <100%> (ø)
... and 7 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b6a11c1...e13f6c1. Read the comment docs.