Build isolation can be a problem with the python-for-android cross compile environment, but disabling it also disables build-system.requires breaking some package installs #6718
Comments
triage-new-issues
bot
added
the
S: needs triage
ghost
changed the title
|
Sorry, I didn't specify something important clearly: obviously I could just make So it's not like this is an unavoidable problem or anything. However, I think this behavior of pip is simply counter-intuitive given what the intended spirit of |
|
Ok, now that I've put in the workaround pip also tries to actually fully build the wheel without |
ghost
changed the title
I may be missing something here (I haven't fully understood the fairly complex example you gave) but surely using |
@pfmoore ok that explains it. In this case, please treat this as a feature request! Why disabled build isolation appears to be required for pip to work right in a cross-compile environment: I tested in python-for-android and pip's build isolation will currently often break in its cross compile environment. The reason is python-for-android needs to preinstall some packages in a different way to make them work, and pip cannot just reinstall them for build isolation, it just doesn't work technically and the install fails if it attempts to do so. Summed up, I don't know what to do! I don't see how this could work
Edit: sorry this context might be missing, I am one of the contributors of https://github.com/kivy/python-for-android making its Edit 2: trimmed this down a little to the core point |
You'd need to explain the request you're making better. (And if it's an issue limited to cross-compiling environments, it would still probably not be high on anyone else's priority list, as I don't believe there are many people working on cross-compilation at the moment - so you may need to provide a PR yourself, or be prepared for a long wait). Currently, pip has two "modes" of operation (described essentially in PEPs 517 and 518, although the implementation details aren't part of the standards). It either does nothing, expecting the user to set up the build environment, and then pip just runs the builds in the global environment, or it isolates individual builds in an environment with the build_dependencies specified in
If you don't, then I doubt anyone else is likely to be able to 😢 Sorry, but this sounds like a pretty specialised problem, and it'll need domain knowledge to know what would solve it. |
|
The problem is pretty simple actually: Is that really so complicated to understand? I think the base problem really is quite simple... Basically I need something like |
ghost
changed the title
|
I renamed the ticket, hopefully that should explain the problem better. It really all comes down again to |
|
So is this just something that has never worked? As far as I am aware, Python has never formally supported cross compilation. If there's something that used to work, but has been broken by recent developments, that's one thing. But if you're trying something that has never worked, and no-one has ever really expected it to, then that's likely to be a much harder problem. (I'm not trying to say that it shouldn't work, or that we don't want to support it, just that it's possible that no-one has ever even looked at this problem before now, and there may be more fundamental issues that you need to solve, rather than assuming that things are expected to work and it's just a matter of getting shallow issues fixed). |
Basically every And the thing is, it makes sense to move away from But to me it looks like this will indirectly cause quite some regressions (for everyone heeding this |
|
Like, just to explain, my own package used Edit: and basically I can change python-for-android in whatever way to make this work, i just don't know how I could unless build isolation is not mandatory for using the new format. Hence this ticket |
|
Did any of this make sense? I'd be up for a text chat or anything to explain all the details of this better if anything is still unclear. This is really a fundamental problem for us at python-for-android I would think (at least I can't see any obvious way to work around it given where pip is heading now) so I really want to help you understand what the issue is so hopefully we at p4a, or you at pip, can find a way to make it work, without all build deps using packages doing nasty hacks or ending up patched by us |
|
I think the feature request here is for something akin to what setup_requires used to do, which was sort of like half build isolation, half not. Currently we have two modes pip can be in:
On the other hand, So I think this feature request is basically to make it possible to still temporarily install things in Is my understanding of what you're asking for correct? |
|
@dstufft yes, that is exactly what would fix our issue. The nicest way in terms of design and user expectation (ignoring what a nightmare that may be to implement) might IMHO be just make pip honor |
Given that this would mean that installs could make arbitrary changes to the user's global package list, that would need to be opt-in, and probably handled very carefully. But that's a detail issue that could be thrashed out as part of a detailed design of a PR covering this. |
|
Oh huh, curious - how would it affect the global environment? We kind of have a locked away virtualenv situation with different cross compilers set, so I'm asking because that sounds like a potential problem if pip suddenly tries to write to the system-wide packages instead of virtualenv site-packages (read-only access wouldn't be an issue, like It also sounds to me like it would be the most intuitive to make Edit: oh I think I get it, you mean |
|
Right, the only way I could see us honoring Whether that's a good idea or not I'm not sure. I'm really loathe to introduce yet another option as I think more options means both more mental overhead to using pip AND it becomes harder to test the combinatorial explosion of possible option configurations and we have more and more interactions untested. I wonder if it would make sense to, in the |
|
@dstufft ok I thought about this for a second, and as I see it, And for use case 1 it would actually be beneficial to not use a temporary environment but keep it around. And use case 2 is something the average user would never do or need, so I think how it behaves in this detail really doesn't matter (since no end user involved that could expect either variant and end up surprised) So to me it makes more sense to not copy Edit: just to explain this better, python-for-android would specify |
|
It's not really possible (or well, reasonable I guess is a better word) to do that, because it's entirely possible that the thing we're building isn't going to end up installed at all. We might build it, and then determine we can't install it and we need to install a different version and throw it away. |
|
@dstufft hm interesting point. Okay that might be unexpected then, I agree. Maybe temporary environment transferred to real site-packages if install goes through then? (Otherwise people expecting to avoid the excessive build time would be kind of left in the rain if even after successful install it's thrown away, I am admittedly also interested in that angle of it myself since this is also a quite big issue with no |
|
Rather than installing things into the user's environment that are neither requested items nor runtime dependencies (because those are the only two things pip ever installs under normal operation), in this new mode pip could report what build dependencies it needed to install, and then the user could manually install them as a one-off extra step? Or maybe this new mode is only ever going to be useful for this one scenario, so installing implicit build dependencies alongside runtime dependencies is OK? I'm just wary of making a general solution less usable by allowing it to have "unusual" side effects. |
|
I'm pretty -1 on pip installing As far as what to do with them in a |
Not sure how that would work for python-for-android unless this is a stable API rather than a text message because otherwise how is python-for-android going to know what to do? And so far pip doesn't really have such an API, right? Because python-for-android would need to tell automatically that it'd need it, otherwise I might as well just make it manually parse
I think there are arguments both for throwing away the temp env later and keeping it around. (Keeping it around because build time saving, throwing it away to keep it clean and potential user expectations) So either I think you just need to pick/decide something or actually add a switch. To me keeping it sounds more useful as a default but I depend so much on the build deps working at all that I'd be really be happy with any outcome here |
|
Note that we do already have a wheel cache, and ideally any wheel we build gets stored in that cache so as long as that cache is working we should (hopefully) only ever build a particular sdist once before we switch to just installing the prebuilt artifact. I think that drastically reduces the usefulness of keeping the temporary directory around. |
|
For what it's worth, this may be relevant: I am in other tickets petitioning for a way to specify source only build dependencies e.g. for So I agree for sure in the long term it doesn't seem that vital to keep it around (Edit: and in the short term it'll not be a disaster not to have it either, so I pretty much meant to say seems fine to just always throw it away) |
chrahunt
added
C: PEP 517 impact
|
I just completed a test today with With this I conclude that I'm fairly certain making |
|
The nice thing about having pyproject.toml be a static file is that you can parse it and carefully manage your build environment manually. It's more work but then your use case is also pretty niche. :) |
|
@pradyunsg I get your point, but in general dealing with pip is already a major pain point in python-for-android, to the point where we had internal discussions if it is worth it at all before taking this new issue into account. The thing is build isolation is great but it comes with its own issues, and the current design pushes it onto us with no real alternative other than even more alternative manual handling Unless you want us to deviate to completely separate packaging things, it would be really nice if there wouldn't be more and more workarounds pushed onto us... there is only so much extra handling code we can reasonably build and maintain to make
So I'd say in overall this is a bit of a worrying view from where we stand. You can see it like that but it simply adds up. The world is already pretty complicated and nobody else in our team probably even understands this issue well enough to be able to make this ticket. There really is a limit to how much we can reasonably do 🤷♀️
So summed up: please don't make us do this. I'd have to explain we have to add even more code most don't know how to maintain... 😢 |
|
@Jonast Ah! I see where you're coming from. Do you have a discuss.python.org account? I think it'd be a good idea to have a broader discussion about how python-for-android uses Python Packaging tooling, and how we could make life easier for y'all! :) |
|
I do, and I wouldn't mind trying to put together a brief overview of how python-for-android currently operates/interacts with pip if anyone is interested (if that is what you had in mind?) - just tell me where to jump in |
|
FWIW, I think we could/should be able to handle things better for projects like Python for Android - it's niche at the moment, maybe, but I'd imagine (and hope!) that it's a growing area. However, nobody outside of the Python for Android community really knows how your processes work, and to my knowledge there's been little validation from either side that the ways you're working are in line with how Python's packaging standards are evolving. So there's a need for flexibility from both sides. At the moment, a lot of the issues you're hitting are coming across as "the way packaging works is broken, because it can't cope with our workflow, and we have to do things this way so either we work around this or you need to fix packaging", and that's not really helping anyone to understand the best way forward (the title of this issue is a good example, "fundamentally incompatible with some cross-compile environments" is pretty alarmist). I understand your frustration, but failing to get your point across will only end up in you being more frustrated. I think there are some basic things that need to be covered as groundwork in any thread that gets started on Discourse.
The alternative is probably that we continue as we are, with individual issues being dealt with on a one-by-one basis, in a way that will feel like a series of workarounds, as there's no focus on the bigger picture. That's probably OK for individual tools, but as you say not for you (and not really for the wider PyPA, as we really don't want to see the packaging community fragmenting the way you suggest could happen). |
I've just brought this back up with the other contributors, and so far there is great interest in this! I will still need a day or two so we can talk about the total basics like how we're internally thinking about forming our opinion when weighing in on things, but other than that this sounds like a really cool idea! I am wondering: documenting how python-for-android works here (in terms of packaging-relevant ways), who the human contact could be if you need a quick response on something, how we possibly want to coordinate things - seems like all this could possibly get a little lost in discourse at some point amidst all the discussions happening. Should we maybe put this info somewhere on our own infrastructure and link it, and update it as discussions go on? Or do you have a good place where you keep your own documentation for how to work with niche groups where this could be added in? (no matter if public or not) Discourse is fine too if you trust it's at least the best place to start with, just thought I would ask Thanks again for offering a deeper communication here, that sounds like a really cool idea! ❤️ |
|
Honestly, there's not much like this in place yet, everything that's available is public, in the PEPs, the distutils-sig archives and in Discourse. So it'll all be very much making things up as we go along. But I'm assuming that the existing processes (of public discussions, PEPs, PEP revisions, etc) will not need much changing, so there's not much needed on the process side. But I would just add a note of caution - Python for Android is still a niche situation. You'll find that most people will have little time or interest in learning the details of how your processes work, so you'll have to do the work of understanding the "normal" workflow assumed by the current PEPs, and explaining the differences as part of proposing any changes. It's not the quickest of processes - not because of lack of interest, just simply because people have very limited time, and other priorities. So you'll need to be patient :-) |
|
I made a topic for this on Discourse. @Jonast My preference is you have a living document containing the overview, that you link to, but really, it's up to you how/where you want to host that overview. I've mentioned it in that post but really, there are no hurries to actually posting that and starting the conversation IMO. https://discuss.python.org/t/packaging-and-python-for-android/2036 |
ghost
mentioned this issue
ghost
changed the title
|
Gave the ticket a less alarmist title as per @pfmoore 's suggestion 🙂 and I will be chiming in on discourse shortly after I gathered some python-for-android developer feedback here: kivy/python-for-android#1932 @pradyunsg living document sounds good I'll write something up and link it on discourse! Thanks again everyone for the opportunity for us to jump into the discussion, that's pretty cool 👍 |
|
I wrote a description now which explains in more detail how python-for-android uses pip: https://github.com/kivy/python-for-android/blob/develop/doc/source/contribute.rst#how-python-for-android-uses-pip which I also linked in the discussion here: https://discuss.python.org/t/packaging-and-python-for-android/2036 I hope this gives some insight for anyone wondering why we (at python-for-android) currently don't know how to get build isolation working, beyond this basic explanation I gave in a comment above. As a result, I hope it's easier to understand why an added feature of being able to turn build isolation off without disabling/skipping |
|
I have rethought all of this with a bit of a break, and currently I am stuck: The discussion so far didn't go much further than "maybe pip should fully support cross compiling" as far as I can see from my side which is an admirable long term goal, but sadly not that helpful with the very current issue of build isolation breaking things. I have now due to the lack of any other workaround pondered the suggestion to manually parse This leaves me so far with no idea how to address this, unless there is a way to disable build isolation where pip itself will still process & install Or does anyone happen to have any other workaround ideas beyond that? Because the only other option is to introduce a python-for-android recipe for my library which I so far managed to avoid, the exact hackish, non-standard workaround which we are trying to reduce in favor of standard packaging tools. (for this I have spent a few months writing code to make it less and less necessary, build isolation kind of being the last roadblock) So this would be quite a step backwards from using standard tooling which would be sad |
|
In your original post, you state:
Could you re-package the patched packages so that they are proper distributions then either keep them in a server (referenced with Edit: Hold on, this is more complicated than that. 😄 |
Well, pip doesn't support dependency cycles in build requirements. A needs B to build and B needs A to build and we only have sources for them -- pip will abort building A and error out. One reasonable restriction that pip had, was requiring build requirements to be available as wheels, so that there's no chance of a build cycle. Otherwise, you'd need to catch cycles.
My suggestion is that you set up an environment containing build-system.requires, before calling pip install. Let's say you have a helper script "pip-wrapper", where the install command is processed as:
Hopefully the above reads as something that's feasible for y'all. I imagine pip-wrapper is a drop in replacement for pip, except it does setup the environment as you'd want it to. |
|
@chrahunt in general we would really want to get rid of all the patching at all, or doing it without pip. However we patch both non-python things and upstream projects that are super Android unaware. Ideally we wouldn't do that anymore or only via pip, but it might take us a long while to get there and I don't know if we ever will, that is one reason I am trying so hard to get this build isolation thing to work better is to avoid a path where we remain with the best option being patching it out of every package using
In any case, things will break if installed in arbitrary order which is what your proposed process doesn't seem to address:
Now if I just look at A and go like well let's start with B as first requirement then things will blow up, because I'd need C around first. So collecting all packages, then collecting all their Edit: oh right I guess with a recursive manual pip download maybe this would work. but that isn't ideal for reasons below
We support almost(all?) types of dependencies right now so this would be a huge regression
I do that in another place but it's a bit not nice since it sometimes gives a wheel, sometimes an archive, which can quite differ in format. And maybe other formats in the future? It's also zipped up and needs to be extracted, so this adds up to a lot of tasks that pip already does, it feels a little like reimplementing pip's functionality again, redundantly, wrapped around pip. It'd be nice to not rely on this too much if there are other ways, but to be fair it does work Edit/small addition: Just to point this out, I personally use both source build dependencies, and chained |
I mean, isn't this covered by just recursion as I suggested? Also, you'd want to be building depth-first in that case, so C (from B<-A) then B (from A) then A in case of that graph. Yea, you'd be redoing some of the archive handling stuff that pip does, yes, but well, I don't have a better suggestion here. It's the least amount of work you'd have to do. |
You can restrict that with --only-binary and --no-binary options. Anyway, I think the default output is what you'd want to consume (just skip the wheel files, since there's no "build" step in them). |
|
Also, IMO, if your main concern now is pip download gives me sources and wheels by default, we have a workable solution. ;) |
|
Well yes what you said would work, I agree. But it would also reimplement some of the archive handling and also either dependency resolution directly, or indirectly via chained wrapper calls. The first seems like a larger but cleaner amount of code maintain (but also kinda more nonsensical to reimplement that) and the last one with a recursive wrapper script confusing to maintain due to the obscured flow of things, on top of all the wrappers we already have. I will try to get my point across a bit better: I am trying to see this from the view of a packaging beginner, from a newbie outside. Parsing a Do my points make any sense here? I am really not objecting to the feasibility of the solution, or that it's too complicated in itself, just that I already have 500 lines of code like that in our project the others probably barely understand and this just looks like yet another thing barely anyone will ever understand later Edit: I also realize this is maybe still the best approach for both projects, or maybe it isn't, I'm not sure yet. I'll sleep about it, I am basically just trying to explain why I'm not that thrilled and unsure about where to go with this Edit 2: I think I can explain better why this is a liability more so than one might think: right now our final install is essentially |
|
Thinking about all this, I think this is likely a case where we just have to accept that your situation is extremely niche, and pip (as a general tool) can't really be expected to cope with it. Build isolation is pip's default mode, and it works well for the vast majority of our users. When it doesn't, we provide an "escape hatch" in It sounds like you have a complex environment management issue, so your solution will be complex. But it's unique to you, so a custom solution is the right option here. It also sounds like you would find it useful if some of pip's internal mechanisms (such as dependency resolution) were available for you to use in developing that solution. It's unfortunate that we don't expose those internal mechanisms, and yes that does make your job harder, but it's the reality right now and we are working on improving that - it's just that it's a complex issue and takes time. So in summary, I think |
They do and I empathize. I second @pfmoore though. |
|
Ok. I understand your position. However, I think calling doing full dependency resolution, especially as complicated and brittle it is right now without a libpip, (and it is, I already implemented it as well as I could and it's not good code) a "certain level of customisation" which we'll "have to write" is a worrying position for us. Because if we are expected to redo dependency resolution and archive extraction, isn't that already half a package manager? And we already have something similar for our own recipes, so we essentially need to maintain this twice. I also find you saying "Build isolation is pip's default mode" worrying because that didn't use to be the case, and it is my impression it isn't even widely adopted now - but if it ever is and if only this is the default which we can no longer work with, what will you add tomorrow and expect us to do as a "certain level of customisation" to keep up then? How many components of pip will we be expected to reimplement from scratch? In conclusion, and this is just my personal opinion, while I can understand your position I also find it of questionable worth to focus our efforts on pip then. Maybe it is long term safer to expand our own independent recipe system into a sort of package manager (which it already is at its basics) and stop chasing after pip, because while it may seem still somewhat feasible now your position doesn't convince me that will be the case tomorrow. This will likely push more effort onto package maintainers and users, but there is only so much we can do. I also understand you would if at all like to solve cross compilation properly one day instead of concessions and extra options for niche cases. That is nice, but if up to that day you want us to rewrite a libpip around pip to keep up then I just don't think this is a good direction for us to go. I'll think more about this and talk to the others, but this is my honest feeling about this right now |
|
This issue is a bit old and maybe the situation evolved. I would be happy to learn about any new development and get pointers to solutions that could have been implemented since 2019. In fredstro/hilbertmodgroup#5 we came up with a similar problem. SageMath is a Python based mathematic library with a lot of non Python softwares and libraries dependencies. On top of SageMath we have user packages such as https://github.com/fredstro/hilbertmodgroup that would ideally be installed via
|
|
As I noted above, this sounds like another specialised case where you might need to build your own solution. In the case of SageMath, I wonder whether the following would work.
There's a bunch of details to work out, and it may not even be suitable for your requirements, but this is the sort of "design your own solution/workflow" approach that I had in mind when I said "a custom solution is the right option here" above. Hopefully it suggests some ideas you can try, if nothing else. |
Environment
latest on pypi right now - installed via -U pip- should be 19.1.1Description
I have run into a case where when I specify --no-build-isolation, pip will install dependencies out of order and if any of these dependencies has a
pyproject.toml, apparently pip may try to obtain wheel metadata of that dependency (by running itssetup.py) without ensuring pyproject.toml'sbuild-system.requirespackages are around. This seems like a bug to me, because isn't the entire point ofbuild-system.requiresthat these packages need to be around for the package to build or even run thesetup.pycorrectly, no matter if the build is isolated or not?I am also quite practically impacted by this bug or behavior because 1. I need to use
pyproject.tomlin dependencies due tosetup_requiresnot liking Cython (the easy_install sandbox messes with the Cython compiler) and I use Cython.pxddeps whichinstall_requiresdoesn't guarantee to be around at build, and 2. I am in a cross compilation environment (python-for-android) where--no-build-isolationis required for all builds because some packages cannot be installed without custom patching, so it's not safe for pip to temporarily reinstall some things as is inherently happening with the build isolationExpected behavior
no
setup.pyis ever run even if just for obtaining wheel metadata without everything inbuild_requiresbeing installed, even if--no-build-isolationis usedHow to Reproduce
I don't have an example package right now, really sorry 😢 (because this all happened in python-for-android) but this should work:
setup.pyin a new folder/my/project/pathpip install git+git://github.com/wobblui/wobblui@562daf6e796fdf8d8a50733b532f9cacae4d4df5(later commits have a workaround!) - wobblui depends onhttps://github.com/JonasT/nettoolsfor building the wheel and specifies this in itspyproject.toml/withbuild-system.requirescd /my/project/path && pip install --user -U --no-build-isolation .wobbluiwheel metadata before it tried to installnettools(which is what happened for me)Output
(really sorry this looks so weird, it's in python-for-android)
^ as you can see it runs wobblui's Cythonization, which happens because wobblui uses
cythonize()to put together its extensions so this automatically runs when thesetup.pyis invoked (see here: https://github.com/wobblui/wobblui/blob/84bc49f34a329b7fccdd2593fba9d91d30379f20/setup.py#L69 ) even if just to get the metadata and not actually build the wheel. This wouldn't be an issue if nettools was installed first as specified here: https://github.com/wobblui/wobblui/blob/84bc49f34a329b7fccdd2593fba9d91d30379f20/pyproject.toml#L2 - which it is in build isolation, but not with--no-build-isolation, apparentlyThe text was updated successfully, but these errors were encountered: