[8970] Improve KubernetesPodOperator guide by tanjinP · Pull Request #9079 · apache/airflow

tanjinP · 2020-05-31T01:19:35Z

Addresses some of #8970

Currently have an outline of the items mentioned (docs/howto/operator/kubernetes.rst) in the issue and looking for feedback if this is a good approach to take to build out this documentation.

There are 5 subheadings, we mention what the content will be, if it includes a code example (currently there are none), and potential references to link (will avoid the blog/guides in favor of official documentation, like Kubernetes)

Make sure to mark the boxes below before creating PR: [x]

Description above provides context of the change
Unit tests coverage for changes (not needed for documentation changes)
Target Github ISSUE in description if exists
Commits follow "How to write a good git commit message"
Relevant documentation is updated including usage instructions.
I will engage committers as explained in Contribution Workflow Example.

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

mik-laj · 2020-05-31T13:33:02Z

docs/howto/operator/kubernetes.rst

-The :class:`~airflow.providers.cncf.kubernetes.operators.kubernetes_pod.KubernetesPodOperator`:
+The :class:`~airflow.providers.cncf.kubernetes.operators.kubernetes_pod.KubernetesPodOperator` allows
+you to create and run Pods on a Kubernetes cluster.
+


Suggested change

.. contents::

:depth: 1

:local:

mik-laj · 2020-05-31T13:34:03Z

Very good direction. We just need it.

mik-laj · 2020-07-03T08:19:08Z

@tanjinP Have you encountered any difficulties? Can I help you?

tanjinP · 2020-07-05T15:07:24Z

@tanjinP Have you encountered any difficulties? Can I help you?

@mik-laj thanks for checking in. Will come back to this and have the initial version complete by end of today if not tomorrow. Will mark the PR as ready then.

…e-k8spodop-guide # Conflicts: # docs/howto/operator/kubernetes.rst

dimberman

@tanjinP this looks fantastic! Just a few suggestions but otherwise thank you for writing this!

dimberman · 2020-07-06T03:36:16Z

docs/howto/operator/kubernetes.rst

The KubernetesPodOperato uses the Kubernetes API to launch a pod in a Kubernetes cluster. By supplying an image URL and a bash command, the operator uses the Kube Python Client...

dimberman · 2020-07-06T03:39:27Z

docs/howto/operator/kubernetes.rst

Users can specify a kubeconfig file using the config_file parameter, otherwise the operator will default to ~/.kube/config

dimberman · 2020-07-06T03:40:34Z

docs/howto/operator/kubernetes.rst

+
+The ``KubernetesPodOperator`` enables task-level resource configuration and is optimal for custom Python
+dependencies that are not available through the public PyPI repository.
+Ultimately, it allows Airflow to act a job orchestrator - no matter the language those jobs are written in.


The ```KubernetesPodOperatoralso allows users to supply a template yaml file using thepod_template_file`` configuration.

dimberman · 2020-07-06T03:42:05Z

docs/howto/operator/kubernetes.rst

To pull images from a private registry*

dimberman · 2020-07-06T03:42:39Z

docs/howto/operator/kubernetes.rst

/airflow/xcom*

Yikes - this was a good one.

tanjinP · 2020-07-07T19:30:26Z

Looks like some static checks failing, will address

mik-laj · 2020-07-08T19:58:04Z

I would be happy to add information that Kubernetes Executor is not required for Kubernetes Pod Operator. Two sentences about the behavior of KubernetesPodOperator when installing into clusters would also be helpful, but we can add them in a separate change.

My opinion is based on the Slack discussion. This is an important feedback on what users are looking for in the documentation.
https://apache-airflow.slack.com/archives/CCV3FV9KL/p1594235643204100

dossett · 2020-07-08T20:19:41Z

docs/howto/operator/kubernetes.rst

 .. note::
  If you use `Google Kubernetes Engine <https://cloud.google.com/kubernetes-engine/>`__, consider
  using the
  :ref:`GKEStartPodOperator <howto/operator:GKEStartPodOperator>` operator as it


This link does currently appear to be working. should it be updated?

Some description that GKEStartPodOperator extends KubernetesPodOperator and that therefore the rest of this documentation is still applicable would be really helpful.

This link works. However, it is not available in the source code, but works in the HTML code generated. Please look at: https://airflow.readthedocs.io/en/latest/howto/operator/kubernetes.html

…e-k8spodop-guide

tanjinP · 2020-07-17T01:27:05Z

@mik-laj

I would be happy to add information that Kubernetes Executor is not required for Kubernetes Pod Operator.

Mentioned this as a note here

Two sentences about the behavior of KubernetesPodOperator when installing into clusters would also be helpful, but we can add them in a separate change.

Not sure what you mean here - what do you mean by installing into cluster? Do you mean setting up the kubeconfig? If so does the section How does this operator work? not cover it?

mik-laj · 2020-07-20T12:55:58Z

Not sure what you mean here - what do you mean by installing into cluster?

I checked my code and this behavior is unique to Cloud Composer. The team made one change to this operator. By default, the kube_config parameter points to a file that contains credentials for the current environment.

tanjinP added 2 commits May 30, 2020 21:08

outline for initial set of items

f862706

small docstring fixes

4b4884b

boring-cyborg bot added area:docs labels May 31, 2020

mik-laj reviewed May 31, 2020

View reviewed changes

mik-laj requested a review from dimberman May 31, 2020 19:33

tanjinP added 4 commits July 5, 2020 11:12

Merge remote-tracking branch 'upstream/master' into issue-8970-improv…

f64e580

…e-k8spodop-guide # Conflicts: # docs/howto/operator/kubernetes.rst

Merge remote-tracking branch 'upstream/master' into issue-8970-improv…

a6ed3c0

…e-k8spodop-guide # Conflicts: # docs/howto/operator/kubernetes.rst

adding examples for docs

518b408

correcting reference marker (~)

e3a933f

tanjinP marked this pull request as ready for review July 6, 2020 02:22

tanjinP requested a review from mik-laj July 6, 2020 02:22

dimberman requested changes Jul 6, 2020

View reviewed changes

addressing comments

d810579

tanjinP changed the title ~~WIP: [8970] Improve KubernetesPodOperator guide~~ [8970] Improve KubernetesPodOperator guide Jul 7, 2020

tanjinP requested a review from dimberman July 7, 2020 02:22

dimberman approved these changes Jul 7, 2020

View reviewed changes

tanjinP added 2 commits July 7, 2020 19:19

resolving static check failures

09db08f

import sort correction (strange that local breeze run didn't catch this)

4fc8f6c

dossett reviewed Jul 8, 2020

View reviewed changes

tanjinP added 2 commits July 16, 2020 20:26

Merge remote-tracking branch 'upstream/master' into issue-8970-improv…

8d8c673

…e-k8spodop-guide

note on executor w/ pod

7148b00

mik-laj merged commit 840799d into apache:master Jul 20, 2020

potiuk mentioned this pull request Jul 27, 2020

Release a 2nd wave of Backport packages #10014

Closed

13 tasks

+.. contents::
+  :depth: 1
+  :local:

Conversation

tanjinP commented May 31, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mik-laj commented May 31, 2020

Uh oh!

mik-laj commented Jul 3, 2020

Uh oh!

tanjinP commented Jul 5, 2020

Uh oh!

dimberman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tanjinP commented Jul 7, 2020

Uh oh!

mik-laj commented Jul 8, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tanjinP commented Jul 17, 2020

Uh oh!

mik-laj commented Jul 20, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Comments

tanjinP commented May 31, 2020 •

edited

Loading