Fix Edge Worker handles orphaned PID files

[OliverWannenwetsch]

Problem description

The Edge Worker run into the following Exception if the worker was shutdown in a undefined way (No clear shutdown).

Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 62, in main
    args.func(args)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 115, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function
    return func(*args, **kwargs)

The root cause is that the current implementation does not consider, if PID file are already existing.
Also it also does not consider if PID files are created by parallel running Edge Worker instances

Bugfix description

This PR contains code to handle existing PID files for Edge workers:

• Orphaned PID files created by crashed instances
• PID files owned by other running Edge worker instances that have configured with the same PID file path (each instance must have its own PID file path!)

Changes in details

• Added function _write_pid_to_pidfile to Edge Worker CLI that handles PID files in their different life cycles
• Added pytests

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[jscheffl]

Thanks for the contribution. That is really a problem I have seen also in my dev if the worker crashed for example if API backend is gone... then the PID file blocked a restart.

Some small requests to improve, otherwise... nice improvement!

[jscheffl]

+1: I see a couple of static checks failing, please run either pre-commit over your changes or breeze static-checks

Note: I'd ignore most of the other errors in the pipeline, seems to be CI general in-stabiltiy)

[jscheffl]

+1... as we are on main but not releasing the package atm because it is not ready, can you please:

• increment the third digit of the version number in providers/src/airflow/providers/edge/provider.yaml
• Add some notes in changelog in providers/src/airflow/providers/edge/CHANGELOG.rst ?

THANKS!

[OliverWannenwetsch]

+1: I see a couple of static checks failing, please run either pre-commit over your changes or breeze static-checks

Note: I'd ignore most of the other errors in the pipeline, seems to be CI general in-stabiltiy)

Fixed in commit 1d0cb9c

[OliverWannenwetsch]

+1... as we are on main but not releasing the package atm because it is not ready, can you please:

• increment the third digit of the version number in providers/src/airflow/providers/edge/provider.yaml
• Add some notes in changelog in providers/src/airflow/providers/edge/CHANGELOG.rst ?

THANKS!

Fixed in commit 33fb875

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.