-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
git checkout could/should be done with --depth=1 #2432
Comments
See: #344 |
@cancan101: thanks for the pointer, I assumed it had to be already discussed somewhere. That PR was withdrawn, with what looks like some confusion as to why - it seems an obvious improvement to that PR would have been to use |
In most uses of pip's git backend, we don't need the full git history - only the target head/branch/tag. This change adds --depth=1 to git fetch and clone commands. NB: "Shallow" checkouts are not supported from arbitrary git commits, so this change removes the ability to install from git SHA1 commit IDs. Tags and branches continue to be supported. Fixes pypa#2432
expect_stderr was added to a few tests due to the following messages: - *warning: --depth is ignored in local clones; use file:// instead.* - *You are in 'detached HEAD' state...* (This is to be expected when the revision shallow-cloned is a tag. As the git-clone manpage says "--branch can also take tags and detaches the HEAD at that commit in the resulting repository.")
yeah seriously it's not fun... +1 to shallow clone! |
This has potential to break setuptools-scm |
potentially) so why not give it a shot, test it with setuptools-scm... pip is already 8, but such an obvious optimization hasn't been included yet... |
+1 Even just having an option or setting to do this would be a huge help. |
@dennybaa a change such as this pretty much completely breaks git describe thus breaking setuptools_scm for certain, which means sucha change would break git for pretty much all packages that use scm metadata on their own or via pbr/setuptools_scm |
@RonnyPfannschmidt Couldn't we just add a |
there is a very intense difference between asking for scm metadata that is local and orchestrating network operations my choice of words intentionally makes this issue seem larger |
@RonnyPfannschmidt Sure, but that's something that only needs to happen once, so more like:
My assumption is that setuptools_scm is called when the package using it is installed. (Is this incorrect?) So if you're using setuptools_scm the behavior would be the same as deep cloning it to begin with; the fetch would occur when network activity is already required. |
From my POV a opt in for shallow clone is just as hard and let's us avoid getting into causing network operations in setuptools-scm |
Workaround for pypa/pip#2432.
@RonnyPfannschmidt if the specific version being shallow cloned was tagged, would setuptools_scm still work? So it would only break if you git install an untagged commit? |
This would definitely be useful as an opt-in flag, such as |
I'd much rather see an opt-out flag. The use cases for opting out are narrow and automated, so why should humans have to pass a flag for optimal behaviour? |
its completely broken behaviour for a growing number of my projects ^^ in future even ones like pytest |
@RonnyPfannschmidt Why would pytest access the git history of a dependency? |
There is a plan to use setuptools-scm for release automation |
I'm all aboard on supporting setuptools-scm. It seems like the ideal solution would be to clone just the commit history but git doesn't seem to have a way to do that yet. In the meantime, what if pip just did a deep clone if |
there is no acceptable clean way to do that atm, and the peps that enable it would need time to grow before pip an implement them |
It might be faster to get a patch merged and released in git than wait for the peps. |
feel free to try then |
To sum up the maintainers' position:
* Crippled because it would be in the hands of users to make sure that their dependencies don't use setuptools-scm when using the flag and one would have to track setuptools-scm dependencies in a separate requirements file. |
true ^^ |
The underlying question here is, what is the fundamental cost of adding an option? One of the foundational pieces of writing for my view point on this comes from Havoc Pennington's Choosing our Preferences which is speaking specifically about GUI applications but they can extend to any kind of tool as well. To go into some more detail though, every option/preference we add has a cost. It costs increased complexity, because every option adds another state that your software can be in, and the more of them you have the more total states your software can be (and this explodes combinatorially for each option you have). It is not unusual at all for software to fail only when multiple specific options are selected because the way the different options interplayed with each other wasn't fully obvious at first. This makes it harder to actually test the software fully because you can quickly get to a point where you have thousands of different possible ways to configure the software. Different options (and features in general) also mean more code, and that additional code needs maintenance over time. Thus for each option, we're increasing the burden of maintaining pip (because each option adds additional code). Another aspect of this is, is since this is something that end users would turn on that individual projects would have no control over, is that this flag would essentially be a "break me" flag for a subset of projects that expect to be able to get a full git version history when they are in a git tree. Thus not only are we adding another option that we have to support and which adds to our combinatorial list of possible states, but now we're also forcing those downstream projects to also deal with that fact (and no matter what, they will get people asking them to support that if pip supports it). Beyond it's direct effect on us though, is the cognitive overhead it places on end users. There is a concept called Hick's Law, which essentially states that the time it takes for a user to make a decision increases logarithmically for every possible option they have in making that decision. That means that for every option we add, we're making it fundamentally harder for someone to use pip, because when they do a If you do any research on some of the fundamentals around designing a good user experience, particularly around options/preferences, one of the largest common themes you'll find is that deciphering when to say Yes and when to say No to a feature is one of the single most important things you can do for the over all user experience of a piece of software... and failing to do that correctly can utterly destroy the usability of your software. Now, with all of that you obviously can't have software with zero options (at least, most software is not the kind of software that is small enough to have zero options) but trying to weigh the impact of adding an option is part of managing a project. In this case it is my belief that the downsides of adding the option in question outweigh the upsides nor do I think the upsides outweigh the upsides of just always doing this. I'm not the only maintainer, so it is entirely possible that one of the other @pypa/pip-committers feels differently to me here (and if they do, they are free to reopen this issue!) but unless someone comes up upsides to this that I have missed (thereby increasing the amount of good it would do) or they come up with a plan (as @pfmoore) to decrease or remove the downsides, I personally am -1 on this. |
@dstufft The said above sounds very theoretical, but though might be true.
Don't use IMO, as pip maintainers:
Besides if things don't break they don't evolve, that's might be even worse for the pip project than that what you describe. But anyways, we all have our opinions. It's just my personal, and I can not insist on anything, just shared to you guys. |
@dennybaa I want this solved as much as anyone. I even wrote a patch implementing the I think you'd be surprised at how many packages a flag would break, and I disagree that we can only progress by breaking things. There have been several proposed solutions in this thread that do not involve fragmenting the python packaging ecosystem. I suggest you pursue one of them instead.
Are you really a dumb user if you don't realize that one of your indirect dependencies uses one of the popular auto-versioning tools? |
@dennybaa Let's just say that the pip maintainers don't agree with how you suggest we treat users confused by an option we add to pip. So we try to avoid adding options that will make using pip a frustrating and confusing experience for our users, no matter what level of experience they have (you may describe them as "dumb", I don't take that view). |
Maybe there's misunderstanding. We've got different opinions. @ryneeverett , @pfmoore I din't call anyone who uses projects providing auto-versioning to python packaging ecosystem "dumb". There was just a list of obvious notions which are my sole opinion. Anyways my bad if you might read it like that...
I was saying that such projects SHOULD document their behavioral specificity rather than pip taking this burden for all. That's why there was number 3) which I mentioned. |
My thought was that as the frontend, pip's job soon will be to just fetch the right requirements and call the backend (to put it dumbly). This would sort of make it more efficient at one specific case. At the end of the day, this is really more of nice-to-have thing. I find this to be good enhancement but also understand that this is a potentially breaking change. I agree with what @pfmoore suggests in #2432 (comment) though so, I'm gonna let this sit. Even I don't like the idea of a flag for it. |
How about rolling this into PEP 518? |
I mean, it sort of makes sense to me but I am wary of like what happened with PEP 426 - we don't want to try and do everything as if we have infinite resources. |
We have a chance to correct build backend behavior in a backwards-compatible way. We should take it. It's clear from this issue that most people (in this issue) want the default to be shallow clones. The reasons for not implementing this are:
The second point doesn't seem accurate; from experience the problem is not resources but getting reviewers to accept your changes. It's also a non-sequitur because the issue was closed. The first point is an intellectually-honest position. However, the practical implications of I think the process forward here should not be to attempt to get this added to a PEP. I think the process forward should be to survey the broader community about whether this should be the default and then if most agree, simply require it for projects with |
How about adding another scheme which will be explicit about intentions ? |
@tzickel |
+1 on this. would be greatly appriciated. |
Just an opinion: version restriction is not needed. Usually projects have a policy that |
That's ridiculous. Major version changes often include non-backward-compatible API changes. There is always going to be a lag time between a project's dependencies being update, and the project updating to use the new version. And it is not true that master must contain a working version, for example Drupal doesn't even HAVE a master branch. |
Incompatible changes are usually announced beforehand and developed in separate branches before merging them into the main branch. And usually it's not that hard to port the software to a new version.
Anyway there exist a branch with the code considered to be bleeding edge. |
Yes, there does. And you're not supposed to use it for production.
Sure. I guess that's why there are still popular python 2 libraries that aren't completely converted to python 3, 8 years after it came out (like twisted).. |
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
pip can install directly from git, great!
When pip installs from git, it downloads the entire history of the git branch - when all it wants is the target version. I believe this can be achieved using the
--depth=1
flag to various git subcommands. The speedup on projects with a long history can be significant.The text was updated successfully, but these errors were encountered: