Concern about PLY dependency being quarantined due to vulnerabilities – blocking Airflow installation

[jskalasariya]

We are currently facing an issue related to the PLY Python package, which is a dependency of Apache Airflow.
Recently, several PyPI packages have been quarantined in our Nexus repository due to known security vulnerabilities. While most of these packages could be resolved by upgrading to a newer minor or patch version, PLY appears to be an exception.

Key points:

• PLY latest available version is 3.11, last released in 2018
• There is no newer version available with vulnerability fixes
• PLY is a required dependency for Airflow, including the latest Airflow versions

Because of this, we are currently unable to install Airflow in environments with strict security policies

Wanted to ask the community:

• Is there any planned effort to replace or remove PLY as a dependency?
• Are there any recommended workarounds or officially supported approaches for handling this in security-restricted environments?
• Has this been discussed previously, or is there an existing proposal we can follow?

Any guidance or direction would be greatly appreciated.

Thanks in advance!

[potiuk]

You can definitely propose a PR removing it.

As far as I could check, howver, ply is NOT a dependency of Airlfow itself, it's a dependency af amazon provider (transitively through jsonpath_ng). You might see it in airflow reference images:

The base reference image:

[jarek:~/code/airflow] main+ 22s 2 ± docker run -it apache/airflow:3.1.7 bash

airflow@5cbc13e2e473:/opt/airflow$ pip freeze | grep ply
ply==3.11
airflow@5cbc13e2e473:/opt/airflow$ exit

The slim image (this one has only preinstalled providers):

[jarek:~/code/airflow] main+ 16s ± docker run -it apache/airflow:slim-3.1.7 bash

airflow@a0b1901320d1:/opt/airflow$ pip freeze | grep ply
airflow@a0b1901320d1:/opt/airflow$ exit

After some quick check with pipdeptree it seems that only Amazon provider uses it (we also use it for some development checks but those do not find it's way into airflow distribution dependencies) . Possibly the amazon team @vincbeck @ferruzzi @o-nikolas might help to discuss finding a replacement for it - I don't think it's a crucial feature of the amazon provider - it has some use #36170 for example added jsonpath_ng.ext.parse support - but I guess this is something that could be easily dropped/replaced.

There are a number of things you can do yourself however to help with that - and it would be great if your company can contribute back by contributing your time (or somoene else to help with it) - since you are interested in removing ply because your coporate rules prevent it, it would also make sense that your company contributes time and effort back to remove it.

So if you would like to spend time and see how to fix it - that would be a welcome contribution - and I am sure the amazon team will be happy to help reviewing and merging it.

For now I see you have several options to proceed:

1. Contribute PR to remove jsonpath_ng from amazon provider
2. Discuss it with the amazon team and convince them to remove it (or find someone else who will)
3. Contact jsonpath_ng project and convince them to remove ply - if they remove it, it wil no longer be a dependency of newer versions of Airflow.
4. You can also contact ply developers and ask them to fix the issues and release a new version

There is also a workaround if you are not using amazon provider:

If you are using airflow reference image, you can instead use airflow slim image and extend it and build your own without amazon provider - thus without jsonpath_ng and ply - or build a completely custom image using our dockerfile - also without amazon provider. The https://airflow.apache.org/docs/docker-stack/build.html has a generic description of various ways how you can extend or customize the image.

So - you are absolutely not blocked, contributions are welcome and it seems that there are some viable paths for you to improve things - so it's only on you now to execute one of those paths @jskalasariya

[o-nikolas]

Hey just checking in on this one. Thanks for raising this concern @jskalasariya

And I can't provide a better answer than what @potiuk already has, those options as next steps are all great. We currently don't have spare cycles to jump on this any time soon, but would happily review a PR.

Can you also open up a Github Issue for this?