Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include airflow_pre_installed_providers.txt into sdist distribution #37388

Merged
merged 1 commit into from Feb 13, 2024

Conversation

Taragolis
Copy link
Contributor

Found that is not possible to build wheel or install airflow from the sdist distribution.

How to reproduce (manual steps)

  1. Build sdist from the main
hatch build -t sdist
───────────────────────────────────────────────────────── sdist ─────────────────────────────────────────────────────────
dist/apache_airflow-2.9.0.dev0.tar.gz

# Save path for the futureexport SDIST_PATH=$(realpath dist/apache_airflow-2.9.0.dev0.tar.gz)
  1. Create new fresh virtual environment, activate and change directory outside of the project, e.g. into the venv directory
TMP_DIR=$(mktemp -d)python -m venv "${TMP_DIR}/venv"cd "${TMP_DIR}"source venv/bin/activatepython -m pip install pip -U
  1. Try to install airflow from sdist or build wheel
pip wheel "${SDIST_PATH}" --no-deps

Processing /Users/taragolis/Projects/common/airflow/dist/apache_airflow-2.9.0.dev0.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [28 lines of output]
      Traceback (most recent call last):
        File "/var/folders/g7/vy97v5m51_v9hdpf_qxqzqvr0000gn/T/tmp.0kwAImfbZo/venv/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/var/folders/g7/vy97v5m51_v9hdpf_qxqzqvr0000gn/T/tmp.0kwAImfbZo/venv/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/var/folders/g7/vy97v5m51_v9hdpf_qxqzqvr0000gn/T/tmp.0kwAImfbZo/venv/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 152, in prepare_metadata_for_build_wheel
          whl_basename = backend.build_wheel(metadata_directory, config_settings)
        File "/private/var/folders/g7/vy97v5m51_v9hdpf_qxqzqvr0000gn/T/pip-build-env-8e6ny3uk/overlay/lib/python3.9/site-packages/hatchling/build.py", line 58, in build_wheel
          return os.path.basename(next(builder.build(directory=wheel_directory, versions=['standard'])))
        File "/private/var/folders/g7/vy97v5m51_v9hdpf_qxqzqvr0000gn/T/pip-build-env-8e6ny3uk/overlay/lib/python3.9/site-packages/hatchling/builders/plugin/interface.py", line 116, in build
          configured_build_hooks = self.get_build_hooks(directory)
        File "/private/var/folders/g7/vy97v5m51_v9hdpf_qxqzqvr0000gn/T/pip-build-env-8e6ny3uk/overlay/lib/python3.9/site-packages/hatchling/builders/plugin/interface.py", line 379, in get_build_hooks
          configured_build_hooks[hook_name] = build_hook(
        File "/private/var/folders/g7/vy97v5m51_v9hdpf_qxqzqvr0000gn/T/pip-build-env-8e6ny3uk/overlay/lib/python3.9/site-packages/hatchling/builders/hooks/custom.py", line 35, in __new__
          hook_class = load_plugin_from_script(path, build_script, BuildHookInterface, 'build_hook')
        File "/private/var/folders/g7/vy97v5m51_v9hdpf_qxqzqvr0000gn/T/pip-build-env-8e6ny3uk/overlay/lib/python3.9/site-packages/hatchling/plugin/utils.py", line 18, in load_plugin_from_script
          spec.loader.exec_module(module)  # type: ignore
        File "<frozen importlib._bootstrap_external>", line 850, in exec_module
        File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
        File "/private/var/folders/g7/vy97v5m51_v9hdpf_qxqzqvr0000gn/T/pip-req-build-d5qqql97/hatch_build.py", line 41, in <module>
          for package in PREINSTALLED_PROVIDERS_FILE.read_text().splitlines()
        File "/Users/taragolis/.pyenv/versions/3.9.10/lib/python3.9/pathlib.py", line 1266, in read_text
          with self.open(mode='r', encoding=encoding, errors=errors) as f:
        File "/Users/taragolis/.pyenv/versions/3.9.10/lib/python3.9/pathlib.py", line 1252, in open
          return io.open(self, mode, buffering, encoding, errors, newline,
        File "/Users/taragolis/.pyenv/versions/3.9.10/lib/python3.9/pathlib.py", line 1120, in _opener
          return self._accessor.open(self, flags, mode)
      FileNotFoundError: [Errno 2] No such file or directory: '/private/var/folders/g7/vy97v5m51_v9hdpf_qxqzqvr0000gn/T/pip-req-build-d5qqql97/airflow_pre_installed_providers.txt'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@Taragolis Taragolis added this to the Airflow 2.8.2 milestone Feb 13, 2024
@potiuk potiuk merged commit 2e15610 into apache:main Feb 13, 2024
56 checks passed
@Taragolis Taragolis deleted the fix-sdist-build branch February 13, 2024 15:46
potiuk pushed a commit that referenced this pull request Feb 13, 2024
potiuk pushed a commit that referenced this pull request Feb 13, 2024
@ephraimbuddy ephraimbuddy added the type:misc/internal Changelog: Misc changes that should appear in change log label Feb 19, 2024
sunank200 pushed a commit to astronomer/airflow that referenced this pull request Feb 21, 2024
ephraimbuddy pushed a commit that referenced this pull request Feb 22, 2024
abhishekbhakat pushed a commit to abhishekbhakat/my_airflow that referenced this pull request Mar 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type:misc/internal Changelog: Misc changes that should appear in change log
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants