PDM can't install packages offline

[SealedServant]

• I have searched the issue tracker and believe that this is not a duplicate.

Make sure you run commands with -v flag before pasting the output.

Steps to reproduce

With the system online, initialize the project like normal to pull in all the packages.

λ pdm install
WARNING: Lockfile does not exist
Updating the lock file...
WARNING: Project requires a python version of >=3.12, The virtualenv is being created for you as it cannot be matched to the right version.
INFO: python.use_venv is on, creating a virtualenv for this project...
Virtualenv is created successfully at C:\Users\xxx\xxx\.venv
Changes are written to pdm.lock.
Synchronizing working set with resolved packages: 96 to add, 0 to update, 0 to remove
...
Installing the project as an editable package...
  ✔ Install xxx 0.1.0 successful

🎉 All complete!

Install pdm-download plugin and attempt to use it to download the packages for offline installation:

λ pdm install --plugins
Plugins are installed successfully into .pdm-plugins.

λ pdm download -v
Error: Client.get() got an unexpected keyword argument 'stream'
(same error message repeated many times)
Error: Client.get() got an unexpected keyword argument 'stream'
0 packages downloaded to packages.

That didn't work, so let's export the requirements.txt.

mkdir .packages
pdm export -o .packages/requirements.txt --no-hashes --no-markers

Now download with pip instead

pdm run pip download -r .packages/requirements.txt -d .packages

For some reason the package paginate does not download as a wheel, so we run pip wheel to make sure all packages are built as wheels to install (otherwise installing with pip later will fail)

pdm run pip wheel -r .packages/requirements.txt -w .packages

Now we've got all the packages for offline. To simulate transferring to offline system, disconnect the internet.

Clean up the irrelevant directories since we need to remake them on the offline system.

rmdir /s /q .venv
rmdir /s /q .coverage
rmdir /s /q .pytest_cache
rmdir /s /q .mypy_cache
rmdir /s /q .ruff_cache
rmdir /s /q .pdm-build
del /f /q .pdm-python
for /r . %%A in (.) do if "%%~nxA"=="__pycache__" rmdir /s /q "%%~fA"

Go to install the packages and recreate the venv on the offline system.

λ pdm install -v
INFO: The saved Python interpreter doesn't match the project's requirement. Trying to find another one.
WARNING: Project requires a python version of >=3.12, The virtualenv is being created for you as it cannot be matched to the right version.
INFO: python.use_venv is on, creating a virtualenv for this project...
Run command: ['C:\\venv\\Scripts\\python.exe', '-m', 'virtualenv', 'C:\\Users\\xxx\\xxx\\.venv', '-p', 'C:\\Program Files\\Python312\\python.EXE', 
'--prompt=xxx-3.12', '--no-pip', '--no-setuptools', '--no-wheel']
created virtual environment CPython3.12.2.final.0-64 in 165ms
  creator CPython3Windows(dest=C:\Users\xxx\xxx\.venv, clear=False, no_vcs_ignore=False, global=False)
  activators BashActivator,BatchActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Virtualenv is created successfully at C:\Users\xxx\xxx\.venv
STATUS: Resolving packages from lockfile...
unearth.preparer: The file . is a local directory, use it directly
pdm.termui: Running PEP 517 backend to get metadata for <Link file:///C:/Users/xxx/xxx (from None)>
pdm.termui: Preparing environment(Isolated mode) for PEP 517 build...
pdm.termui: ======== Start resolving requirements ========
pdm.termui:   pdm-backend
pdm.termui:   python==3.12.2
pdm.termui:   Adding requirement pdm-backend
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\venv\Bin\pdm.exe\__main__.py", line 7, in <module>
  File "C:\venv\Lib\site-packages\pdm\core.py", line 358, in main
    return core.main(args or sys.argv[1:])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\pdm\core.py", line 276, in main
    raise cast(Exception, err).with_traceback(traceback) from None
  File "C:\venv\Lib\site-packages\pdm\core.py", line 271, in main
    self.handle(project, options)
  File "C:\venv\Lib\site-packages\pdm\core.py", line 207, in handle
    command.handle(project, options)
  File "C:\venv\Lib\site-packages\pdm\cli\commands\install.py", line 100, in handle
    actions.do_sync(
  File "C:\venv\Lib\site-packages\pdm\cli\actions.py", line 238, in do_sync
    synchronizer.synchronize()
  File "C:\venv\Lib\site-packages\pdm\installers\synchronizers.py", line 395, in synchronize
    to_add, to_update, to_remove = self.compare_with_working_set()
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\pdm\installers\synchronizers.py", line 215, in compare_with_working_set
    candidates = self.candidates.copy()
                 ^^^^^^^^^^^^^^^
  File "C:\Program Files\Python312\Lib\functools.py", line 995, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\pdm\installers\synchronizers.py", line 151, in candidates
    if self.should_install_editables():
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\pdm\installers\synchronizers.py", line 169, in should_install_editables
    metadata = self.self_candidate.prepare(self.environment).metadata
               ^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python312\Lib\functools.py", line 995, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\pdm\installers\synchronizers.py", line 139, in self_candidate
    return self.environment.project.make_self_candidate(not self.no_editable)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\pdm\project\core.py", line 521, in make_self_candidate
    can.prepare(self.environment).metadata
  File "C:\venv\Lib\site-packages\pdm\models\candidates.py", line 610, in metadata
    result = self.prepare_metadata()
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\pdm\models\candidates.py", line 514, in prepare_metadata
    return self._get_metadata_from_build(self._unpacked_dir, metadata_parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\pdm\models\candidates.py", line 592, in _get_metadata_from_build
    self._metadata_dir = builder(source_dir, self.environment).prepare_metadata(metadata_parent)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\pdm\builders\editable.py", line 21, in prepare_metadata
    self.install(self._requires, shared=True)
  File "C:\venv\Lib\site-packages\pdm\builders\base.py", line 296, in install
    install_requirements(missing, env)
  File "C:\venv\Lib\site-packages\pdm\installers\core.py", line 28, in install_requirements
    resolved, _ = resolve(resolver, reqs, environment.python_requires, max_rounds=resolve_max_rounds, keep_self=True)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\pdm\resolver\core.py", line 39, in resolve
    result = resolver.resolve(requirements, max_rounds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\resolvelib\resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\resolvelib\resolvers.py", line 397, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "C:\venv\Lib\site-packages\resolvelib\resolvers.py", line 173, in _add_to_criteria
    if not criterion.candidates:
           ^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\resolvelib\structs.py", line 127, in __bool__
    next(iter(self))
         ^^^^^^^^^^
  File "C:\venv\Lib\site-packages\resolvelib\structs.py", line 136, in __iter__
    self._factory() if self._iterable is None else self._iterable
    ^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\pdm\resolver\providers.py", line 226, in matches_gen
    candidates = self._find_candidates(reqs[0])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\pdm\resolver\providers.py", line 196, in _find_candidates
    return self.repository.find_candidates(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\pdm\models\repositories.py", line 174, in find_candidates
    cans = LazySequence(self._find_candidates(requirement, minimal_version=minimal_version))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\pdm\models\repositories.py", line 436, in _find_candidates
    for c in finder.find_all_packages(requirement.project_name, allow_yanked=requirement.is_pinned)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\unearth\finder.py", line 310, in find_all_packages
    self._find_packages(package_name, allow_yanked), hashes=hashes or {}
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\unearth\finder.py", line 290, in _find_packages
    return sorted(all_packages, key=self._sort_key, reverse=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\unearth\collector.py", line 178, in collect_links_from_location
    yield from _collect_links_from_index(session, location)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\unearth\collector.py", line 198, in _collect_links_from_index
    page = fetch_page(session, location)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\unearth\collector.py", line 185, in fetch_page
    resp = _get_html_response(session, location)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\unearth\collector.py", line 220, in _get_html_response
    resp = session.get(
           ^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\httpx\_client.py", line 1054, in get
    return self.request(
           ^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\httpx\_client.py", line 827, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\httpx\_client.py", line 914, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\httpx\_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\httpx\_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\httpx\_client.py", line 1015, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\hishel\_sync\_transports.py", line 197, in handle_request
    regular_response = self._transport.handle_request(request)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\venv\Lib\site-packages\httpx\_transports\default.py", line 232, in handle_request
    with map_httpcore_exceptions():
  File "C:\Program Files\Python312\Lib\contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "C:\venv\Lib\site-packages\httpx\_transports\default.py", line 86, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [Errno 11001] getaddrinfo failed

Try again, but just make the venv manually and include pip so that pip can install the packages.

λ   pdm venv create --with-pip
Virtualenv C:\Users\xxx\xxx\.venv is created successfully
λ   pdm run pip install -r .packages/requirements.txt --no-index -f .packages
Looking in links: .packages
Processing ... (truncated)
Installing collected packages: ... (truncated)
Successfully installed ... (truncated)

Whew, okay, we were able to manually create the venv and get the packages installed with pip. PDM should be able to handle the rest.

I should be able to run pdm install now and it will detect that we are offline and that the packages from the lockfile are already installed in the virtual environment. We need it to still install my project as an editable dependency.

λ pdm install 
See C:\Users\xxx\AppData\Local\pdm\pdm\Logs\pdm-install-w_u289cr.log for detailed debug log.
[ConnectError]: [Errno 11001] getaddrinfo failed
WARNING: Add '-v' to see the detailed traceback

No dice. I really need to be able to install the project as editable. pip cannot do it because it fails to find pdm-backend since we are offline.

Let's try pdm lock --check or pdm sync instead.

λ pdm lock --check

λ pdm sync
See C:\Users\xxx\AppData\Local\pdm\pdm\Logs\pdm-install-srwrj0na.log for detailed debug log.
[ConnectError]: [Errno 11001] getaddrinfo failed
WARNING: Add '-v' to see the detailed traceback

Bummer. Maybe pdm fix?

λ pdm fix
No problem is found, nothing to fix.

Actual behavior

Try to use pdm commands offline but most of them fail.

Expected behavior

I expected:

• pdm-download to work
• pdm to be able to automatically install packages it finds in the .packages folder without an internet connection
• pdm to be able to automatically detect packages already installed manually into the virtual environment and install the project as editable like it normally would with an internet connection

Environment Information

# Paste the output of `pdm info && pdm info --env` below:
λ pdm info && pdm info --env
PDM version:
  2.15.0
Python Interpreter:
  C:\Users\xxx\xxx\.venv\Scripts\python.exe (3.12)
Project Root:
  C:/Users/xxx/xxx
Local Packages:

{
  "implementation_name": "cpython",
  "implementation_version": "3.12.2",
  "os_name": "nt",
  "platform_machine": "AMD64",
  "platform_release": "10",
  "platform_system": "Windows",
  "platform_version": "10.0.19045",
  "python_full_version": "3.12.2",
  "platform_python_implementation": "CPython",
  "python_version": "3.12",
  "sys_platform": "win32"
}

[frostming]

• pdm download doesn't pull the build dependencies(pdm-backend) which are not in the lockfile
• pdm export doesn't pin build dependencies either for the same reason
• pip wheel doesn't know the current project you want to install as editable

To solve this, you can add pdm-backend as a dependency, repeat the same steps, and install offline with pdm install --no-isolation

BTW, Error: Client.get() got an unexpected keyword argument 'stream' this should be fixed by pdm-download@0.1.3

[SealedServant]

@frostming, thank you for the quick response. Lightning fast bug fix on pdm-download mate⚡

Is it intended behavior for pdm-download to download all the package binaries for every platform? pip seems to download just the wheels for the current platform.

Notes

Adding pdm-backend to the project dependencies works to get it downloaded as a wheel (in .packages\pdm_backend-2.2.1-py3-none-any.whl), but pip cannot seem to find pdm-backend in the venv when I try to install the project as editable.

$ pdm venv create --with-pip
Virtualenv C:\Users\xxx\xxx\.venv is created successfully
INFO: Virtualenv C:\Users\xxx\xxx\.venv is reused.

$  pdm run pip install -r .packages/requirements.txt --no-index -f .packages
Looking in links: .packages
Processing ...
Installing collected packages: ...
Successfully installed ... 

$  pdm run pip install -e .
Looking in indexes: https://xxx:****@xxx.xxx.xxx.xxx/repository/pypi-proxy/simple/
Obtaining file:///C:/Users/xxx/xxx
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [8 lines of output]
      Looking in indexes: https://xxx:****@xxx.xxx.xxx.xxx/repository/pypi-proxy/simple/
      WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x0000016E815A1EE0>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed')': /repository/pypi-proxy/simple/pdm-backend/
      WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x0000016E81608920>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed')': /repository/pypi-proxy/simple/pdm-backend/
      WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x0000016E81608B60>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed')': /repository/pypi-proxy/simple/pdm-backend/
      WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x0000016E81608D70>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed')': /repository/pypi-proxy/simple/pdm-backend/
      WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x0000016E81608F80>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed')': /repository/pypi-proxy/simple/pdm-backend/
      ERROR: Could not find a version that satisfies the requirement pdm-backend (from versions: none)
      ERROR: No matching distribution found for pdm-backend
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

I tried using pdm install --no-isolation instead, but it still appears that pdm tries to connect to the internet and fails.

C:\Users\xxx\xxx(dev -> origin)
λ pdm install -v --no-isolation
STATUS: Resolving packages from lockfile...
unearth.preparer: The file . is a local directory, use it directly
pdm.termui: Running PEP 517 backend to get metadata for <Link file:///C:/Users/xxx/xxx (from None)>
pdm.termui: Preparing environment(Non-isolated mode) for PEP 517 build...
pdm.termui: ======== Start resolving requirements ========
pdm.termui:   editables
pdm.termui:   python==3.12.2
pdm.termui:   Adding requirement editables
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\pdm\Bin\pdm.exe\__main__.py", line 7, in <module>
  File "C:\pdm\Lib\site-packages\pdm\core.py", line 358, in main
    return core.main(args or sys.argv[1:])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\pdm\core.py", line 276, in main
    raise cast(Exception, err).with_traceback(traceback) from None
  File "C:\pdm\Lib\site-packages\pdm\core.py", line 271, in main
    self.handle(project, options)
  File "C:\pdm\Lib\site-packages\pdm\core.py", line 207, in handle
    command.handle(project, options)
  File "C:\pdm\Lib\site-packages\pdm\cli\commands\install.py", line 100, in handle
    actions.do_sync(
  File "C:\pdm\Lib\site-packages\pdm\cli\actions.py", line 238, in do_sync
    synchronizer.synchronize()
  File "C:\pdm\Lib\site-packages\pdm\installers\synchronizers.py", line 395, in synchronize
    to_add, to_update, to_remove = self.compare_with_working_set()
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\pdm\installers\synchronizers.py", line 215, in compare_with_working_set
    candidates = self.candidates.copy()
                 ^^^^^^^^^^^^^^^
  File "C:\Program Files\Python312\Lib\functools.py", line 995, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\pdm\installers\synchronizers.py", line 151, in candidates
    if self.should_install_editables():
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\pdm\installers\synchronizers.py", line 169, in should_install_editables
    metadata = self.self_candidate.prepare(self.environment).metadata
               ^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python312\Lib\functools.py", line 995, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\pdm\installers\synchronizers.py", line 139, in self_candidate
    return self.environment.project.make_self_candidate(not self.no_editable)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\pdm\project\core.py", line 521, in make_self_candidate
    can.prepare(self.environment).metadata
  File "C:\pdm\Lib\site-packages\pdm\models\candidates.py", line 610, in metadata
    result = self.prepare_metadata()
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\pdm\models\candidates.py", line 514, in prepare_metadata
    return self._get_metadata_from_build(self._unpacked_dir, metadata_parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\pdm\models\candidates.py", line 592, in _get_metadata_from_build
    self._metadata_dir = builder(source_dir, self.environment).prepare_metadata(metadata_parent)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\pdm\builders\editable.py", line 24, in prepare_metadata
    self.install(requires)
  File "C:\pdm\Lib\site-packages\pdm\builders\base.py", line 296, in install
    install_requirements(missing, env)
  File "C:\pdm\Lib\site-packages\pdm\installers\core.py", line 28, in install_requirements
    resolved, _ = resolve(resolver, reqs, environment.python_requires, max_rounds=resolve_max_rounds, keep_self=True)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\pdm\resolver\core.py", line 39, in resolve
    result = resolver.resolve(requirements, max_rounds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\resolvelib\resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\resolvelib\resolvers.py", line 397, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "C:\pdm\Lib\site-packages\resolvelib\resolvers.py", line 173, in _add_to_criteria
    if not criterion.candidates:
           ^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\resolvelib\structs.py", line 127, in __bool__
    next(iter(self))
         ^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\resolvelib\structs.py", line 136, in __iter__
    self._factory() if self._iterable is None else self._iterable
    ^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\pdm\resolver\providers.py", line 226, in matches_gen
    candidates = self._find_candidates(reqs[0])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\pdm\resolver\providers.py", line 196, in _find_candidates
    return self.repository.find_candidates(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\pdm\models\repositories.py", line 174, in find_candidates
    cans = LazySequence(self._find_candidates(requirement, minimal_version=minimal_version))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\pdm\models\repositories.py", line 436, in _find_candidates
    for c in finder.find_all_packages(requirement.project_name, allow_yanked=requirement.is_pinned)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\unearth\finder.py", line 310, in find_all_packages
    self._find_packages(package_name, allow_yanked), hashes=hashes or {}
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\unearth\finder.py", line 290, in _find_packages
    return sorted(all_packages, key=self._sort_key, reverse=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\unearth\collector.py", line 178, in collect_links_from_location
    yield from _collect_links_from_index(session, location)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\unearth\collector.py", line 198, in _collect_links_from_index
    page = fetch_page(session, location)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\unearth\collector.py", line 185, in fetch_page
    resp = _get_html_response(session, location)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\unearth\collector.py", line 220, in _get_html_response
    resp = session.get(
           ^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\httpx\_client.py", line 1054, in get
    return self.request(
           ^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\httpx\_client.py", line 827, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\httpx\_client.py", line 914, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\httpx\_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\httpx\_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\httpx\_client.py", line 1015, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\hishel\_sync\_transports.py", line 197, in handle_request
    regular_response = self._transport.handle_request(request)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\httpx\_transports\default.py", line 232, in handle_request
    with map_httpcore_exceptions():
  File "C:\Program Files\Python312\Lib\contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "C:\pdm\Lib\site-packages\httpx\_transports\default.py", line 86, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [Errno 11001] getaddrinfo failed

[frostming]

@SealedServant did you set the index to the local path, which is similar to pip's -f option?

https://arc.net/l/quote/zjcoxfvv

[SealedServant]

Is there a way to call pdm install -f like with pip? Or perhaps to set the local location in pyproject.toml?

[[tool.pdm.source]]
name = "local"
url = "./packages"
type = "find_links"

[frostming]

Or perhaps to set the local location in pyproject.toml?

You can use the file url:

[[tool.pdm.source]]
name = "local"
url = "file:///path/to/packages"
type = "find_links"

[SealedServant]

Thanks @frostming, I will try this out on Monday and report back the results here. Will file URL work with relative path like so?

[[tool.pdm.source]]
name = "local"
url = "file:///.packages"
type = "find_links"

I have a private pypi repo set up in pyproject.toml that I am hoping doesn't interfere with adding this secondary source. This is to ensure all packages come from our pypi mirror rather than externally.

[[tool.pdm.source]]
name = "pypi"
url = "https://our.pypi.mirror"
verify_ssl = true
include_package = ["*"]

Some comments:

I think the .packages directory should be a temporary/transient place in the project dir to store the wheels for offline installation, so I am not sure if permanently being in the pyproject.toml is the best approach.

I made a pdm download command with this behavior:

[tool.pdm.scripts]
download = { shell = """
  mkdir .packages \
  pdm export -o .packages/requirements.txt --no-hashes --no-markers \
  pdm run pip wheel -r .packages/requirements.txt -w .packages
""" }

I envision a pdm install --no-index -f .packages so that pdm will do everything that we expect like generating/checking/syncing with the lock file, installing the current project as editable, et cetera, but it won't fail if it can't reach the configured sources.

Ultimately, I think the best approach would be if we could just move the entire project directory, which is the intent of the user. However, too many tools such as virtualenv, venv, editables, are creating hardcoded absolute file paths (!!!) and which totally break when you move the project location. This is not the fault of pdm by any means, but users suffer from these other tools not enabling portable virtual environments. If pdm fix could somehow detect broken paths in places like .venv, .pdm-build, .pdm-python and fix them, that would make it so that we don't have to inefficiently download and install everything to site-packages again just for the purpose of recreating the .venv so that it works. But this might be out of scope of pdm and understandable that we just need to work with such limitations.

[SealedServant]

[[tool.pdm.source]]
name = "local"
url = "file:///${PROJECT_ROOT}/.packages"
type = "find_links"

Running pdm install --no-isolation offline is giving the same error as before, unfortunately. I also tried with url set to file:///./.packages

Also seeing this when downloading the packages:

WARNING: Location 'file:///${PROJECT_ROOT}/packages/' is ignored: it is neither a file nor a directory.

[frostming]

Stick to absolute paths for now

Also seeing this when downloading the packages:

Where does this come from, neither pdm nor unearth emits that

[frostming]

Closed by dfa6a8d

[SealedServant]

Stick to absolute paths for now

Also seeing this when downloading the packages:

Where does this come from, neither pdm nor unearth emits that

It looks like it is being emitted by pip when I have the local source url in the pyproject.toml after running this command in my project root to get the wheels for offline installation:

$ pdm run pip wheel -r .packages/requirements.txt -w .packages
Looking in indexes: https://xxx:xxx@xxx.com/repository/pypi-proxy/simple/
Looking in links: file:///${PROJECT_ROOT}/.packages
WARNING: Location 'file:///${PROJECT_ROOT}/.packages' is ignored: it is neither a file nor a directory.

pyproject.toml look like this:

[build-system]
requires = ["pdm-backend"]
build-backend = "pdm.backend"

# ...

[tool.pdm]
distribution = true

[[tool.pdm.source]]
name = "pypi"
url = "https://${USERNAME}:${PASSWORD}@xxx.xxx.xxx/repository/pypi-proxy/simple/"
verify_ssl = true
include_packages = ["*"]

[[tool.pdm.source]]
name = "local"
url = "file:///${PROJECT_ROOT}/.packages" # do I need to set this env var manually or does PDM know this is my project?
type = "find_links"

Installation offline fail with following message:

$ pdm install --no-isolation
WARNING: Project requires a python version of <3.13,>=3.12, The virtualenv is being created for you as it cannot be matched to the right version.
INFO: python.use_venv is on, creating a virtualenv for this project...
Virtualenv is created successfully at C:\Users\xxx\xxx\.venv
See C:\Users\xxx\AppData\Local\pdm\pdm\Logs\pdm-install-nyi26u0v.log for detailed debug log.
[ConnectError]: [Errno 11001] getaddrinfo failed
WARNING: Add '-v' to see the detailed traceback

[frostming]

It looks like it is being emitted by pip

it makes no sense to me either. by no means pip reads the config under tool.pdm

[SealedServant]

It looks like it is being emitted by pip

it makes no sense to me either. by no means pip reads the config under tool.pdm

Found this: https://stackoverflow.com/questions/43046544/unusual-error-while-installing-any-pip-based-package

I think I know why, the requirements.txt exported by pdm is including the indices at the bottom. Also note that it is including the sys_platform for select packages:

# This file is @generated by PDM.
# Please do not edit it manually.

-----(truncated)-----
ghp-import==2.1.0
griffe==0.44.0
httpx==0.27.0
identify==2.5.36
mypy-extensions==1.0.0
nodeenv==1.8.0
-----(truncated)-----
pywin32==306; sys_platform == "win32"
-----(truncated)-----
text-unidecode==1.3
tomlkit==0.12.4
types-python-dateutil==2.9.0.20240316
watchfiles==0.21.0
wcwidth==0.2.13
websockets==12.0
zipp==3.18.1
--index-url https://${USERNAME}:${PASSWORD}@xxx.xxx.xxx/repository/pypi-proxy/simple/
--find-links file:///${PROJECT_ROOT}/.packages

[frostming]

fine. it makes sense. now no need to complain anymore and try the main branch with pdm-download

[SealedServant]

fine. it makes sense. now no need to complain anymore and try the main branch with pdm-download

Sorry @frostming. I really like pdm and I am trying to avoid using poetry if I can help it 😄 Thanks for all the time you have put into pdm and helping people get issues resolved.

I tried the main branch and I tried coding the absolute path to the .packages folder with url = "file:///C:/Users/.../.../.packages" just to make sure it would work.

I am not sure why, but pdm install --no-isolation is still failing offline. It appears that termui / unearth are still trying to connect to pypi to obtain setuptools (also editables and pdm-backend) despite the fact that I have added these packages to the project dependencies in pyproject.toml, the local .packages folder, and pdm itself with command pdm self add pdm-backend editables setuptools.

I appears that pdm is not attempting to resolve the additional sources in the pyproject.toml and stops after the first source resolution failure.

unearth.collector: Collecting links from https://***@***.***.***.***/repository/pypi-proxy/simple/setuptools/
unearth.auth: Found credentials in url for ***.***.***.***
pdm.termui: Error occurs
Traceback (most recent call last):
  File "C:\pdm\Lib\site-packages\httpx\_transports\default.py", line 69, in map_httpcore_exceptions
    yield
  File "C:\pdm\Lib\site-packages\httpx\_transports\default.py", line 233, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\httpcore\_sync\connection_pool.py", line 216, in handle_request
    raise exc from None
  File "C:\pdm\Lib\site-packages\httpcore\_sync\connection_pool.py", line 196, in handle_request
    response = connection.handle_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\httpcore\_sync\connection.py", line 99, in handle_request
    raise exc
  File "C:\pdm\Lib\site-packages\httpcore\_sync\connection.py", line 76, in handle_request
    stream = self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\httpcore\_sync\connection.py", line 122, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\httpcore\_backends\sync.py", line 205, in connect_tcp
    with map_exceptions(exc_map):
  File "C:\Program Files\Python312\Lib\contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "C:\pdm\Lib\site-packages\httpcore\_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: [Errno 11001] getaddrinfo failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\pdm\Lib\site-packages\pdm\termui.py", line 257, in logging
    yield logger
  File "C:\pdm\Lib\site-packages\pdm\cli\actions.py", line 238, in do_sync
    synchronizer.synchronize()
  File "C:\pdm\Lib\site-packages\pdm\installers\synchronizers.py", line 445, in synchronize
    handlers[kind](key, progress)
  File "C:\pdm\Lib\site-packages\pdm\installers\synchronizers.py", line 282, in install_candidate
    self.manager.install(can)
  File "C:\pdm\Lib\site-packages\pdm\installers\manager.py", line 33, in install
    prepared.build(),
    ^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\pdm\models\candidates.py", line 402, in build
    self._obtain(allow_all=False)
  File "C:\pdm\Lib\site-packages\pdm\models\candidates.py", line 438, in _obtain
    self.link = _find_best_match_link(
                ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\pdm\models\candidates.py", line 105, in _find_best_match_link
    found = attempt_to_find()
            ^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\pdm\models\candidates.py", line 96, in attempt_to_find
    best = finder.find_best_match(req.as_line(), hashes=hashes).best
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\unearth\finder.py", line 391, in find_best_match
    best_match = next(iter(applicable_candidates), None)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\unearth\finder.py", line 324, in _find_packages_from_requirement
    yield from self._find_packages(requirement.name, allow_yanked)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\unearth\finder.py", line 290, in _find_packages
    return sorted(all_packages, key=self._sort_key, reverse=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\unearth\collector.py", line 178, in collect_links_from_location
    yield from _collect_links_from_index(session, location)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\unearth\collector.py", line 198, in _collect_links_from_index
    page = fetch_page(session, location)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\unearth\collector.py", line 185, in fetch_page
    resp = _get_html_response(session, location)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\unearth\collector.py", line 220, in _get_html_response
    resp = session.get(
           ^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\httpx\_client.py", line 1054, in get
    return self.request(
           ^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\httpx\_client.py", line 827, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\httpx\_client.py", line 914, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\httpx\_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\httpx\_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\httpx\_client.py", line 1015, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\hishel\_sync\_transports.py", line 197, in handle_request
    regular_response = self._transport.handle_request(request)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\pdm\Lib\site-packages\httpx\_transports\default.py", line 232, in handle_request
    with map_httpcore_exceptions():
  File "C:\Program Files\Python312\Lib\contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "C:\pdm\Lib\site-packages\httpx\_transports\default.py", line 86, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [Errno 11001] getaddrinfo failed

[frostming]

you need to completely ban pypi offline by setting the name of the local source to "pypi" instead of "local"

[SealedServant]

@frostming, thank you for all the support. I was finally able to get everything to work seamlessly. I want to take some notes here for other users who might find this in the future. It would be nice if the documentation could have notes on using PDM offline, maybe in the advanced usage section.

Full permission granted to copy/paste/modify this for PDM documentation.

Use PDM in Offline Environments

Prerequisites

You must ensure that your build dependencies are in your pyproject.toml.

For example, if you are using pdm-backend like so

[build-system]
requires = ["pdm-backend"]
build-backend = "pdm.backend"

then the following dependencies are required to use pdm-backend offline with the --no-isolation flag

[tool.pdm.dev-dependencies]
build = ["editables>=0.5", "pdm-backend>=2.3.0"]

Download the Dependencies

On an online system, ensure the PDM project is initialized normally:

pdm install

Make a directory .packages:

mkdir .packages

Export the dependencies to .packages/.requirements.txt:

pdm export --format requirements --output .packages/.requirements.txt --no-hashes --no-markers

Download the dependency wheels into the directory:

pdm run pip wheel --requirement .packages/.requirements.txt --wheel-dir .packages 

Download PDM

On an online system, create a virtual environment to install pdm into:

python -m venv .pdm

Activate the virtual environment with the platform-specific script located in .pdm/Scripts and install pdm to the virtual environment:

pip install pdm

Make a directory .pdm-packages:

mkdir .pdm-packages

Export the dependencies to .pdm-packages/.requirements.txt:

pip freeze > .pdm-packages/.requirements.txt

Download the pdm dependencies as wheels into the directory:

pip wheel --requirement .pdm-packages/.requirements.txt --wheel-dir .pdm-packages 

Optionally, perform a git clean to prepare the project for the offline system. Make sure to exclude the .packages and .pdm-packages directory.

WARNING: git clean -fdx will remove any files not tracked by git and any files included in .gitignore, such as a .venv inside the project. Since virtual environments are not portable, it is recommended to recreate it on the offline system.

git clean -fdx --exclude .packages --exclude .pdm-packages

Install PDM

On the offline system, install pdm to a directory of your choice. It is recommended to install pdm into a virtual environment.

With your virtual environment activated, install the packages from .pdm-packages:

pip install -r .pdm-packages/.requirements.txt --no-index --find-links .pdm-packages

Once pdm and it's dependencies are installed to the virtual environment, it is recommended to make a symlink to the pdm executable and add it to your system path.

Install the Dependencies

On the offline system, ensure that pdm is installed and available on the system PATH.

In the project directory you have transferred, make sure that pip is installed:

  pdm run python -m ensurepip --default-pip

Install the dependencies:

pdm run pip install -r .packages/.requirements.txt --no-index --find-links .packages

Check the project file for changes, update the lock file if needed, sync with the lock file, and install the project as editable:

pdm install --no-isolation