Add nodePort declaration to chart/values.schema.json #26945

joseph-max-coalfire · 2022-10-07T22:42:50Z

closes: #26812
Adding a nodePort declaration to the chart/values.schema.json file to allow users to explicitly define exactly which port the nodePort setting should use when deploying with the Helm chart.

This is important for users who have external load balancers and firewalls/security groups that have predefined traffic ports for Airflow. (e.g. a load balancer is expecting the airflow-webserver to listen on port 31000, and it's set up via Terraform, so I'd rather be able to specify a known port in Terraform beforehand rather than have to grab a new port every time I build Airflow)

Tested locally on minikube with the following values.yaml override.

webserver:
    service:
      type: NodePort
      ports:
        - name: airflow-ui
          port: 80
          targetPort: airflow-ui
          nodePort: 31000

Cluster comes up successfully with airflow-webserver service listening on expected nodePort. Output of the deployed airflow-webserver service object's YAML below:

Name:                     airflow-webserver
Namespace:                default
Labels:                   app.kubernetes.io/managed-by=Helm
                          chart=airflow-1.7.0-dev
                          component=webserver
                          heritage=Helm
                          release=airflow
                          tier=airflow
Annotations:              meta.helm.sh/release-name: airflow
                          meta.helm.sh/release-namespace: default
Selector:                 component=webserver,release=airflow,tier=airflow
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.104.151.191
IPs:                      10.104.151.191
Port:                     airflow-ui  80/TCP
TargetPort:               airflow-ui/TCP
NodePort:                 airflow-ui  31000/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Also tested without specifying a nodePort option (defaulting to ClusterIP) to ensure new schema doesn't break backwards compatibility.

boring-cyborg · 2022-10-07T22:42:53Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

dstandish · 2022-10-08T02:39:28Z

Do you think you could add a test for this?

joseph-max-coalfire · 2022-10-09T19:38:13Z

Do you think you could add a test for this?

@dstandish just added one and tested locally. Let me know if it needs more coverage

dstandish · 2022-10-10T21:31:19Z

tests/charts/test_webserver.py

+            )
+        ]
+    )
+    def test_nodeport_service(self, ports, expected_ports):


looks like your params aren't consistent with those provided in expand
but let's see what the tests tell us
i'd also add an entry (in expand) for when no nodePort is provided

looks like your params aren't consistent with those provided in expand

@dstandish Not sure I follow-- I used the test_ports_overrides param structure as a reference, and it looks like CI checks are passing

i'd also add an entry (in expand) for when no nodePort is provided

Added (lines 730-733) and tested locally with success. We'll see if the remote CI checks pass as well

all good.

one thing though.

looking here, it looks like name is not a valid attr for nodeport. probably best to remov that.

joseph-max-coalfire · 2022-10-17T22:19:09Z

@dstandish would you be able to re-review this this week? No worries if not, just checking. It also looks like I somehow set you as the only reviewer, so if there's someone else who can review it as well, let me know

joseph-max-coalfire · 2022-10-18T13:04:49Z

^ added a trailing comma that was causing the Black CI check to fail. Tests can be re-run when possible

…t setting of a nodePort port value when deploying via Helm chart

…till specified as NodePort

potiuk · 2022-11-07T21:41:00Z

I run the workflow. @dstandish ?

dstandish

small things

dstandish · 2022-11-08T06:00:06Z

tests/charts/test_webserver.py

+            )
+        ]
+    )
+    def test_nodeport_service(self, ports, expected_ports):


all good.

one thing though.

looking here, it looks like name is not a valid attr for nodeport. probably best to remov that.

tests/charts/test_webserver.py

Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

joseph-max-coalfire · 2022-11-08T23:42:05Z

Thanks @dstandish and @potiuk!

@dstandish I've reviewed and committed your recommended changes

boring-cyborg · 2022-11-14T00:57:50Z

Awesome work, congrats on your first merged pull request!

joseph-max-coalfire requested review from dstandish and jedcunningham as code owners October 7, 2022 22:42

boring-cyborg bot added the area:helm-chart Airflow Helm Chart label Oct 7, 2022

joseph-max-coalfire force-pushed the add-nodeport-to-helm-schema branch from 0ebfafe to de9562e Compare October 10, 2022 20:51

dstandish reviewed Oct 10, 2022

View reviewed changes

joseph-max-coalfire force-pushed the add-nodeport-to-helm-schema branch 5 times, most recently from b031585 to 52d3aa9 Compare October 12, 2022 20:19

joseph-max-coalfire requested review from dstandish and removed request for jedcunningham October 12, 2022 22:03

joseph-max-coalfire force-pushed the add-nodeport-to-helm-schema branch 2 times, most recently from 44a5a0b to 24e04a7 Compare October 17, 2022 16:49

joseph-max-coalfire force-pushed the add-nodeport-to-helm-schema branch 2 times, most recently from 047ca0c to 75a38eb Compare October 18, 2022 13:03

joseph-max-coalfire force-pushed the add-nodeport-to-helm-schema branch 6 times, most recently from 23162de to ee252ad Compare October 24, 2022 15:06

potiuk force-pushed the add-nodeport-to-helm-schema branch from ee252ad to 41feda7 Compare October 31, 2022 01:47

Joseph Morris added 2 commits November 1, 2022 10:25

Add nodePort declaration to chart/values.schema.json to allow explici…

278e27a

…t setting of a nodePort port value when deploying via Helm chart

Add test for NodePort service type for webserver component

773fe00

Joseph Morris added 4 commits November 1, 2022 10:25

Add NodePort test case where no NodePort is provided but service is s…

fb27f0f

…till specified as NodePort

Add missing comma to test_nodeport_service parameters for style purposes

b8e2dc3

Update test_nodeport_service unit test with new pytest format

90606c7

Black formatting for tests/charts/test_webserver.py

6f1b237

joseph-max-coalfire force-pushed the add-nodeport-to-helm-schema branch from 41feda7 to 6f1b237 Compare November 1, 2022 15:25

dstandish approved these changes Nov 8, 2022

View reviewed changes

joseph-max-coalfire and others added 2 commits November 8, 2022 13:13

Remove unnecessary parentheses from nodeport test

892a43a

Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

Remove "name" value from nodeport test

92c07ee

Co-authored-by: Daniel Standish <15932138+dstandish@users.noreply.github.com>

potiuk approved these changes Nov 14, 2022

View reviewed changes

potiuk merged commit cc571e8 into apache:main Nov 14, 2022

jedcunningham mentioned this pull request Jan 31, 2023

Status of testing of Apache Airflow Helm Chart 1.8.0rc1 #29273

Closed

28 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add nodePort declaration to chart/values.schema.json #26945

Add nodePort declaration to chart/values.schema.json #26945

joseph-max-coalfire commented Oct 7, 2022

boring-cyborg bot commented Oct 7, 2022

dstandish commented Oct 8, 2022

joseph-max-coalfire commented Oct 9, 2022 •

edited

dstandish Oct 10, 2022 •

edited

joseph-max-coalfire Oct 10, 2022 •

edited

dstandish Nov 8, 2022

joseph-max-coalfire commented Oct 17, 2022

joseph-max-coalfire commented Oct 18, 2022

potiuk commented Nov 7, 2022

dstandish left a comment

dstandish Nov 8, 2022

joseph-max-coalfire commented Nov 8, 2022

boring-cyborg bot commented Nov 14, 2022

Add nodePort declaration to chart/values.schema.json #26945

Add nodePort declaration to chart/values.schema.json #26945

Conversation

joseph-max-coalfire commented Oct 7, 2022

boring-cyborg bot commented Oct 7, 2022

dstandish commented Oct 8, 2022

joseph-max-coalfire commented Oct 9, 2022 • edited

dstandish Oct 10, 2022 • edited

Choose a reason for hiding this comment

joseph-max-coalfire Oct 10, 2022 • edited

Choose a reason for hiding this comment

dstandish Nov 8, 2022

Choose a reason for hiding this comment

joseph-max-coalfire commented Oct 17, 2022

joseph-max-coalfire commented Oct 18, 2022

potiuk commented Nov 7, 2022

dstandish left a comment

Choose a reason for hiding this comment

dstandish Nov 8, 2022

Choose a reason for hiding this comment

joseph-max-coalfire commented Nov 8, 2022

boring-cyborg bot commented Nov 14, 2022

joseph-max-coalfire commented Oct 9, 2022 •

edited

dstandish Oct 10, 2022 •

edited

joseph-max-coalfire Oct 10, 2022 •

edited