Skip to content

Remove unused [datalib] extra from OpenAI provider base install#69408

Open
YAshhh29 wants to merge 3 commits into
apache:mainfrom
YAshhh29:remove-openai-datalib-extra
Open

Remove unused [datalib] extra from OpenAI provider base install#69408
YAshhh29 wants to merge 3 commits into
apache:mainfrom
YAshhh29:remove-openai-datalib-extra

Conversation

@YAshhh29

@YAshhh29 YAshhh29 commented Jul 5, 2026

Copy link
Copy Markdown

The apache-airflow-providers-openai package pulls in the [datalib]
extra of the openai SDK — which transitively installs numpy, pandas,
and pandas-stubs — for every user of this provider. That extra has no
functional use inside this provider today, so it is pure bloat and CVE
noise for anyone installing the provider.

Investigation

I audited every code path that could depend on numpy/pandas reaching
users through this provider:

Check Location Result
Provider source providers/openai/src/**/*.py 0 references to numpy/pandas/DataFrame
Provider unit tests providers/openai/tests/unit/** 0 references
Provider example DAGs / system tests providers/openai/tests/system/openai/example_*.py 0 references — only pendulum + Airflow imports
Provider docs providers/openai/docs/** Only auto-generated dep tables
Operator return types operators/openai.py list[float], str, str | None — no DataFrames
Cross-provider consumers (RAG examples) providers/{pgvector,pinecone,weaviate}/tests/system/**/example_*_openai.py pgvector & pinecone: 0 references. weaviate has its own pandas dep declared in providers/weaviate/pyproject.toml
Historical usage git log providers/openai/pyproject.toml [datalib] was carried forward across every major version bump (openai 0.x → 1.x → 2.x); recent PR #69068 just bumped openai[datalib]>=1.66.0openai[datalib]>=2.37.0 without justifying the extra
Upstream SDK docs openai/openai-python README (SDK 2.x) [datalib] is not mentioned anywhere in the modern SDK docs ([aiohttp], [bedrock], [realtime] are — [datalib] is not)

Why it was there in the first place

[datalib] was the extra used by OpenAI SDK 0.x's legacy CLI
(openai tools fine_tunes.prepare_data), which processed JSONL
fine-tuning files via pandas. That CLI was removed when OpenAI SDK 1.x
shipped in 2023, but the extra remained in the package for backward
compatibility. This Airflow provider carried the extra forward across
every subsequent version bump without re-evaluating whether it was
still needed — a classic transitive fossil dependency.

What changes

  • providers/openai/pyproject.toml: base dep is now openai>=2.37.0 (no [datalib]); inline comment records the rationale so it doesn't get re-added by the next version bump.
  • providers/openai/README.rst and providers/openai/docs/index.rst: dep table updated to reflect the base dep.
  • providers/openai/docs/changelog.rst: behavior note under the Changelog header explaining the change and pointing users at pip install 'openai[datalib]' if they had unknowingly relied on the transitive numpy/pandas install for their DAG code.

Impact for users

  • Everyone who installs the OpenAI provider stops pulling in numpy, pandas, and pandas-stubs transitively — smaller Docker images, smaller pip-audit surfaces.
  • Users of OpenAIHook, OpenAIEmbeddingOperator, OpenAIResponseOperator, and the batch operators are unaffected — none of their code paths use pandas/numpy.
  • The one edge case is a user whose DAG code (not the provider) does import pandas and was relying on this provider to bring pandas along transitively. The changelog note points those users at the exact one-line fix.

@eladkal eladkal requested a review from kaxil July 6, 2026 18:37
@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants