Added ability to enable/disable scheduler and webserver

[kajanajustin]

Adding the ability to enable and disable the scheduler and webserver if you just want to run just workers in kubernetes. This change adds the enabled boolean in both the scheduler and webserver sections of the values.yaml file

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[potiuk]

I am not sure we want it. If you reallly want to do worker-only deployment, you should create your own helm/resources -that will be way simpler and our chart is not meant to be as versatile to support such cases. cc: @jedcunningham @ephraimbuddy - WDYT?

[jedcunningham]

@potiuk I've actually come around to this idea over time. You can sorta do it today with scaling those components to 0, but you end up with all that stuff in k8s anyways. Plus it also can help with the worker queues problem (#34219) - not super gracefully, mind you, but better than roll your own certainly, as you can just use multiple values files to keep it sane.

I don't see this as being materially different than some of the other advanced stuff we expose to our users already.

[potiuk]

@potiuk I've actually come around to this idea over time. You can sorta do it today with scaling those components to 0, but you end up with all that stuff in k8s anyways. Plus it also can help with the worker queues problem (#34219) - not super gracefully, mind you, but better than roll your own certainly, as you can just use multiple values files to keep it sane.

I don't see this as being materially different than some of the other advanced stuff we expose to our users already.

Fine for me then - if we are ready to support those cases :). I actually like it if we do it because this might help in some of the future descriptions of multi-tenant deployment. I initially thought that we do not want to make anything about Hel Chart with multitenancy, but actually now that you wrote it - it seems pretty plausible for anyone doing future multi-tenancy setup to do it by composing multiple charts. So yeah. I see how it can be useful if we want to support it.

It would be good however maybe in this case to add a bit of documentation about that case - for example describe a few deployments (like the multiple queue case) to explain users what our intention is with those parameters. @kajanajustin - would you be open to add a short description of potential use cases for those parameters ? Maybe in "Production Guide" part of the docs: https://airflow.apache.org/docs/helm-chart/stable/production-guide.html ?

[jedcunningham]

We also need test coverage for this 👍.

[kajanajustin]

@potiuk @jedcunningham Let me know what you guys think about the doc edits I made. Let me know if I need to add any additional descriptions

[potiuk]

Minimal, but good enough for me :) . Still a few tests would be needed to approve that one.

[kajanajustin]

@potiuk anything else needed from me at this time? I assume you guys have the testing handled from here

[potiuk]

@potiuk anything else needed from me at this time? I assume you guys have the testing handled from here

No - unit tests are always part of the PRs. We have separately run tests for the helm chart in helm_tests folder which are basically testing that when you set the right parameter, the resources are produced properly after passing throuhg helm.

Our contributig guide is currently not properly rendered by github, but I have a PR in progress where all the contributing docs ( are nicely split to smaller docs, so you can see the docs nicely rendering here https://github.com/apache/airflow/blob/refactor-contributing-docs/contributing-docs/testing/helm_unit_tests.rst and you can look at other tests in helm_tests and add your tests in similar way.

[potiuk]

also you have static checks failing and you should fix them. For that you can take a look at this split doc in my PR https://github.com/apache/airflow/blob/refactor-contributing-docs/contributing-docs/08_static_code_checks.rst

[hussein-awala]

Looks good, just need a test per resource to ensure that it is not created when the enabled value is false.

[kajanajustin]

@hussein-awala how do we go about testing that?

[potiuk]

@hussein-awala how do we go about testing that?

That's what I described in #36991 (comment)

[kajanajustin]

@hussein-awala @potiuk added additional unit tests, let me know if there's anything else you guys need to see

[kajanajustin]

@potiuk @jedcunningham static checks and the test_can_be_disabled for the scheduler were failing so I added a new class for the function. Let me know if you think everything looks good so we can merge this in

[amoghrajesh]

Generally looks good, wondering if we should add tests to the other k8s objects that are deployed if shceduler / webserver flags are turned on.

[kajanajustin]

@jedcunningham what else needs to be updated here? I see some checks are still failing anything needed from me to merge this?

[jedcunningham]

I'm pretty sure those failures aren't related to your change, but I need to look into a little closer before I can ignore them. Also, we need to handle the comment discussion.

[potiuk]

Rebvased it. Those were issues connected with yesterday's "google-core-api" relase brekaing compatibility (we workarounded it but also they fixed it this morning with 2.16.1 release after we raised the bug.

[kajanajustin]

@jedcunningham comment has been removed can we merge this in

[kajanajustin]

@potiuk @jedcunningham dont want this to get stale and want to use this on the next helm chart release.

Can we merge this in? Looks like everyone has approved who had comments before

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

[jedcunningham]

Thanks @kajanajustin! Congrats on your first commit 🎉

[kajanajustin]

@jedcunningham thank you for all your help!