Scripts/systemd weren't updated to Airflow 3

[radumas]

Apache Airflow version

3.0.1

If "Other Airflow 2 version" selected, which one?

No response

What happened?

The documentation (https://airflow.apache.org/docs/apache-airflow/stable/howto/run-with-systemd.html) suggests using the scripts in the below folder for setting up with SystemD

https://github.com/apache/airflow/tree/main/scripts/systemd

However, they have not been updated to Airflow 3 and they do not start the dag-processor or triggerer, and start the webserver instead of the api-server

What you think should happen instead?

No response

How to reproduce

Follow the step in the documentation

These files can be used as-is by copying them over to /usr/lib/systemd/system.

Operating System

RHEL

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

No response

Anything else?

We can try to submit a PR to correct this, but honestly at the moment our in-place Airflow 3 upgrade on RHEL hasn't been working super well, so I'm not sure we're the best placed to be making this update (though I think this is mostly issues with configs and not systemd unit files).

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[sjyangkevin]

It looks like the script is out-dated. I also recently observed some changes between the 2.x docker-compose file and 3.x docker-compose. Also, in the script folder, haven’t found anything related with database initialization. At high level, it can be helpful to see how these scripts are used to deploy airflow. How it is different from the docker-compose file. 🤔 and wondering if it is a deprecated process since the code is not updated for a while.

[potiuk]

Yes - if you want to update the scripts either of you - feel free - either of you. We are not actively checking or testting systemd setup but if someone does and would like to update the scripts based on their setup, it's a good idea.

[radumas]

Question: It seems like the initial systemd files assume that airflow is installed at the system level, instead of in a virtual environment.

For example:

airflow/scripts/systemd/airflow-scheduler.service

Line 29 in 2eb420c

ExecStart=/bin/airflow scheduler

whereas in our service file we have:

ExecStart= bash -c 'source /home/airflow/airflow_venv/bin/activate ; /home/airflow/airflow_venv/bin/airflow scheduler'

Which is activating the virtual environment (and loading any variables configured there) and then starting the scheduler.

Any preferences?

I could update the documentation to be more explicit about the airflow installation location and what to modify if the install is within a virtual environment.

Also, we don't run flower, kerberos or worker, so I can't touch those with authority if there's something to change there.

[sjyangkevin]

I want to put my two cents that might be worth to consider. From my understanding, your case may be to deploy and run Airflow as a standalone instance. flower and worker more relate to if you want to use CeleryExecutor and want to have a service to monitor the workers. From the service files, looks it assumes a running instance of Postgres, MySQL, or Redis, on the same machine. I think it can also be helpful to manage Python dependencies in the Airflow environment through the virtual environment, but if the machine only runs the single Airflow instance, probably it’s not necessary. Feel free to correct me if I am wrong on any aspect.

[radumas]

I'm in agreement with that @sjyangkevin. From your knowledge, do the Flower & Worker files look like they would need updating to Airflow 3?

@potiuk, does the scripts folder in GitHub get included in the installation? I'm wonder if I should also update the documentation to point people more explicitly at where they find that folder, since the references don't include links back to GitHub.

airflow/airflow-core/docs/howto/run-with-systemd.rst

Lines 26 to 28 in 241144a

	In the ``scripts/systemd`` directory, you can find unit files that
	have been tested on Redhat based systems. These files can be used as-is by copying them over to
	``/usr/lib/systemd/system``.