Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Git dependencies are re-cloned every time some commands run #1370

Closed
gjoseph92 opened this issue Sep 7, 2022 · 0 comments · Fixed by #1374
Closed

Git dependencies are re-cloned every time some commands run #1370

gjoseph92 opened this issue Sep 7, 2022 · 0 comments · Fixed by #1374
Labels
⭐ enhancement Improvements for existing features

Comments

@gjoseph92
Copy link

Is your feature request related to a problem? Please describe.

Some commands re-clone git dependencies even when it feels like they shouldn't have to:

  • pdm install --dry-run --check
  • pdm export

I would have thought these commands could run without cloning the git repo, or making any network requests at all. These are commands I would expect to always run fast (they would both be reasonable to use in pre-commit hooks).

[project]
name = ""
version = ""
description = ""
authors = [
    {name = "Gabe Joseph", email = "gjoseph92@gmail.com"},
]
dependencies = [
    "scheduler-profilers @ git+https://github.com/gjoseph92/scheduler-profilers.git@6b51132fdc622c23fd17ac1be3d0d131b793bee0",
]
requires-python = ">=3.9"
license = {text = "MIT"}

[build-system]
requires = ["pdm-pep517>=1.0.0"]
build-backend = "pdm.pep517.api"

[tool]
[tool.pdm]
» pdm install --dry-run --check -v
unearth: Cloning https://github.com/gjoseph92/scheduler-profilers.git (revision: 6b51132fdc622c23fd17ac1be3d0d131b793bee0) to /var/folders/rs/wdnmv5lj02x7sh19rg3nyfyr0000gn/T/pdm-build-om34njdi
unearth: Resolved https://github.com/gjoseph92/scheduler-profilers.git to commit 6b51132fdc622c23fd17ac1be3d0d131b793bee0

» pdm export -v > /dev/null
The exported requirements file is no longer cross-platform. Using it on other platforms may cause unexpected result.
unearth: Cloning https://github.com/gjoseph92/scheduler-profilers.git (revision: 6b51132fdc622c23fd17ac1be3d0d131b793bee0) to /var/folders/rs/wdnmv5lj02x7sh19rg3nyfyr0000gn/T/pdm-build-n5voavm6
unearth: Resolved https://github.com/gjoseph92/scheduler-profilers.git to commit 6b51132fdc622c23fd17ac1be3d0d131b793bee0

Describe the solution you'd like

These commands do not make any network requests and are always fast.

Even if I have a git ref like git+https://github.com/foo/bar.git@main, not pointing at a specific commit but at a branch, export shouldn't have to resolve what commit main is currently pointing to—it should just use whatever revision has already been locked into the lockfile. For example, using the above pyproject.toml but replacing @6b51132fdc622c23fd17ac1be3d0d131b793bee0 with @main, my lockfile contains:

[[package]]
name = "scheduler-profilers"
version = "0.3.0"
requires_python = ">=3.7.1,<4.0"
git = "https://github.com/gjoseph92/scheduler-profilers.git"
ref = "main"
revision = "6b51132fdc622c23fd17ac1be3d0d131b793bee0"
summary = "Profile the dask distributed scheduler with py-spy or viztracer"
dependencies = [
    "distributed>=2.30.0",
    "py-spy<1.0.0,>=0.3.9",
    "viztracer<1.0.0,>=0.12.3",
]

Since the revision is already locked, why does it need to be re-resolved?

More broadly, with git dependencies, I wouldn't expect the dependency to ever be cloned except when doing pdm add or pdm update. As long as the lockfile is pinned to a commit, that commit should continue to be used until the user explicitly asks to change it (via pdm update). If we know we're going to continue using that commit (regardless of whether main has changed upstream), why re-clone the repo?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
⭐ enhancement Improvements for existing features
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant