Get rid of pyarrow-hotfix for CVE-2023-47248 (#36697)

The #35650 introduced a hotfix for Pyarrow CVE-2023-47248. So far
we have been blocked from removing it by Apache Beam that limited
Airflow from bumping pyarrow to a version that was not vulnerable.

This is now possible since Apache Beam relesed 2.53.0 version on
4th of January 2023 that allows to use non-vulnerable pyarrow.

We are now bumping both Pyarrow and Beam minimum versions to
reflect that and remove pyarrow hotfix.

airflow/providers/apache/beam/provider.yaml

@@ -52,7 +52,9 @@ versions:
 
 dependencies:
   - apache-airflow>=2.6.0
-  - apache-beam>=2.47.0
+  # Apache Beam > 2.53.0 and pyarrow > 14.0.1 fix https://nvd.nist.gov/vuln/detail/CVE-2023-47248.
+  - apache-beam>=2.53.0
+  - pyarrow>=14.0.1
 
 integrations:
   - integration-name: Apache Beam

generated/provider_dependencies.json

@@ -56,7 +56,8 @@
   "apache.beam": {
     "deps": [
       "apache-airflow>=2.6.0",
-      "apache-beam>=2.47.0"
+      "apache-beam>=2.53.0",
+      "pyarrow>=14.0.1"
     ],
     "cross-providers-deps": [
       "google"

setup.py

@@ -351,10 +351,6 @@ def write_version(filename: str = str(AIRFLOW_SOURCES_ROOT / "airflow" / "git_ve
 otel = ["opentelemetry-exporter-prometheus"]
 pandas = [
     "pandas>=0.17.1",
-    # Use pyarrow-hotfix to fix https://nvd.nist.gov/vuln/detail/CVE-2023-47248.
-    # We should remove it once Apache Beam frees us to upgrade to pyarrow 14.0.1
-    "pyarrow-hotfix",
-    "pyarrow>=9.0.0",
 ]
 password = [
     "bcrypt>=2.0.0",