New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for noarch conda packages? #86

Open
ceball opened this Issue Jul 2, 2017 · 47 comments

Comments

Projects
None yet
@ceball

ceball commented Jul 2, 2017

https://docs.continuum.io/anaconda-repository/admin/noarch-packages includes this:

NOTE: Noarch packages are not compatible with Anaconda constructor. If you intend to use the packages with Anaconda constructor, build the packages for specific operating systems.

I couldn't find an issue about this here, so I wanted to check: is this still the case?

@mingwandroid

This comment has been minimized.

Collaborator

mingwandroid commented Jul 2, 2017

If it is we should fix it.

@ceball

This comment has been minimized.

ceball commented Jul 5, 2017

Sorry, I missed this near the end of the README:

Constructor does not work with noarch-Python packages. All conda packages must be available for the platform you are building the installer for.

In that case my question is: can this be a feature request, or is it not part of the plan?

Thanks

@csoja csoja added the type-feature label Jul 6, 2017

@ericdill

This comment has been minimized.

Contributor

ericdill commented Sep 20, 2017

@mingwandroid @csoja is noarch support going to be added to constructor?

@parente

This comment has been minimized.

parente commented Sep 20, 2017

To add a concrete example: If I add a norarch package to my constructor definition today, it builds fine but upon running it, all the noarch packages wind up in <env root>/site-packages instead of in <env root>/lib/python3.6/site-packages (or whatever version of Python I've specified in the constructor). As a result, they're not in the python path and not importable. In contrast, when I conda install a noarch package, it winds up in <env root>/lib/python<version>/site-packages.

Perhaps a path prefix missing somewhere in the noarch case?

@mingwandroid

This comment has been minimized.

Collaborator

mingwandroid commented Sep 20, 2017

I've been cautioning people not to use noarch: python for a while now. It's just not ready for prime-time I'm afraid and this is a good example of that (others include permissions with .pyc files, menu shortcuts .. the list probably goes on and on).

It'd be nice to get some of this fixed, contributions are welcome.

@parente

This comment has been minimized.

parente commented Sep 20, 2017

Interesting. We're just starting to dabble in the noarch space and I, at least, was not aware there was a litany of issue.

Happy to try to contribute a fix for this initial problem here, but no one should wait on me if they get to it first.

@ericdill

This comment has been minimized.

Contributor

ericdill commented Sep 20, 2017

@mingwandroid are you recommending against the use of noarch: python in general or just in the context of constructor?

@mingwandroid

This comment has been minimized.

Collaborator

mingwandroid commented Sep 20, 2017

I personally wouldn't use it until all of the issues are fixed. It's a bit chicken and egg though, I mean if no one uses it then no one will run into the problems and attempt to fix them. I'm not even sure what can be done about .pyc files in locked down scenarios.

Everyone's use-case is different though. With my Anaconda Distro hat on I have to take a very conservative approach.

@mbargull

This comment has been minimized.

Member

mbargull commented Sep 20, 2017

@mingwandroid I've a strong interest to push noarch: python support generally. Would you mind if we open up some meta "noarch: python issues"-issue over at https://github.com/conda/conda/issues where we compose a list of problems you/we see with the current noarch: python-support. We could then tackle each problem in separate issues/tests/PRs -- I'm quite willing to contribute to this!
(For menu shortcuts we have conda/conda#5153 which could be solved if ContinuumIO/menuinst#48 gets traction.)

@danielfrg

This comment has been minimized.

danielfrg commented Sep 20, 2017

This is actually an issue now that conda-forge is starting to push some pkgs with noarch: python.

Right now I think its impossible to create a working constructor based on new pkgs from conda-forge, for example six wont work now that it has noarch: python. Is there a way to tell constructor to completely ignore noarch pkgs?

@mingwandroid

This comment has been minimized.

Collaborator

mingwandroid commented Sep 20, 2017

@danielfrg, not to my knowledge. As a team we're all a bit too busy to fix this. The fix would seem to necessarily intrude heavily into the solver code which would be probably be unpleasant for all involved.

@mbargull, sure, sounds good and thanks for helping out (as usual). Same thing though, we're trying to get the next release out so are all too busy to do anything for this at present. I'll have some time to think about it in a couple of weeks, but I'd love to get your take on how to approach it. I am leaning towards taking the core installer part of conda out of conda (and Python actually) and adding low-level support that would be useful for menuinst to that. This could include conversion from e.g. .png (or .svg?) to platform specific icons.

@danielfrg

This comment has been minimized.

danielfrg commented Sep 20, 2017

I just tested this on constructor=1.3 that doesn't support noarch pkgs and it worked fine by just pulling from linux-64 for example.

I am not sure why was this introduced in #43 if it actually doesn't work, i guess it work for noarch pkgs that are not python libs but most of them are noarch-python pkgs so it break most of the time. And it actually doesn't break, like other people reported the installer finishes but libs are impossible to import.

I guess the solution would be for a flag to remove the noarch path of the channel (https://github.com/conda/constructor/blob/master/constructor/fcp.py#L181)

Even better would be to actually support noarch-python pkgs but I am afraid i don't understand enough to know what the problem is. Is it the postlink script of the noarch-python pkgs?

@mingwandroid

This comment has been minimized.

Collaborator

mingwandroid commented Sep 20, 2017

We've moved back to using conda in constructor now instead of libconda (unfortunately we're working out of a different branch for reasons I will not bore you with: https://github.com/conda/constructor/tree/cross_conda_interface), but yes, since they're in different subdirs, it's probably quite easy to stop conda from picking them up, cc'ing @kalefranz?

The work for un-noarch-ifying noarch python packages is done in conda at present.

@mingwandroid

This comment has been minimized.

Collaborator

mingwandroid commented Sep 20, 2017

The core problem with constructor is that it has 3 backends (macOS & linux shell script, macOS pkg and Windows NSIS) and they do not share very much in common.

Also these 3 bits of installation logic are not the same code as what conda itself runs.

This is why I want to extract the core package installation logic into a (fully static) standalone executable (that is bundled with the installers), then each of the 3 installer front-ends do little more than run this executable.

@danielfrg

This comment has been minimized.

danielfrg commented Sep 21, 2017

Got it. Thanks for explaining that, i know there is a lot of context i dont have. Hopefully there is an easy fix for having constructor working again with conda-forge now that they have some noarch pkgs. other than manually listing everysingle version on the constructor, that is really not that bad.

ericdill added a commit to ericdill/constructor that referenced this issue Sep 21, 2017

Blacklist noarch python packages
Constructor is not yet compatible with noarch python packages as per the
discussion in conda#86

This "fix" allows us to continue using constructor like we have been
previously before noarch: python packages started appearing. When the
fixes described in conda#86 land this fix can be reverted.
@ericdill

This comment has been minimized.

Contributor

ericdill commented Sep 21, 2017

@mingwandroid One short-term fix is just to blacklist all noarch: python packages from constructor. This is a pretty simple fix as you can see in #115. I think this would be a nice short-term solution so that we can still continue to use constructor like we have been until constructor fully supports the noarch python packages. Thoughts?

ericdill added a commit to ericdill/constructor that referenced this issue Sep 21, 2017

@mingwandroid

This comment has been minimized.

Collaborator

mingwandroid commented Sep 21, 2017

I'm fine with it, we may need @ilanschnell to review and merge it though (presumably you'd need a release/tag with this change?), unless can target our new branch where I'm happy to review and merge it.

@ericdill

This comment has been minimized.

Contributor

ericdill commented Sep 21, 2017

presumably you'd need a release/tag with this change?

Selfishly, this patch means that I've already figured out what I need to fix and build internally so that I can continue using constructor at my day-job.

Putting my open-source contributor hat on, it'd be a shame to not release/tag with this update since, without it, anyone that is currently using constructor 1.5.5 and targeting the conda-forge channel will start producing "bad" constructor installer scripts if they unknowingly pull in a noarch python package. I'd like to suggest branching at 1.5.5, applying this patch and releasing a 1.5.6.

unless can target our new branch where I'm happy to review and merge it.

What's the new branch? Are you going to be cutting releases from that new branch?

What's the deal with the 1.6.0 tag? I don't see that version available anywhere?

ericdill added a commit to ericdill/constructor that referenced this issue Sep 21, 2017

Blacklist noarch python packages
Constructor is not yet compatible with noarch python packages as per the
discussion in conda#86

This "fix" allows us to continue using constructor like we have been
previously before noarch: python packages started appearing. When the
fixes described in conda#86 land this fix can be reverted.
@mingwandroid

This comment has been minimized.

Collaborator

mingwandroid commented Sep 21, 2017

Here is the branch: https://github.com/conda/constructor/tree/cross_conda_interface

I don't know the answers to the rest of your questions, I'll try to bring them up with people who might.

@ericdill

This comment has been minimized.

Contributor

ericdill commented Sep 25, 2017

@mingwandroid any updates here? Would be lovely to get #115 in and a new version made for the folks starting to use noarch packages

@mingwandroid

This comment has been minimized.

Collaborator

mingwandroid commented Sep 25, 2017

I'm on vacation, pinging @csoja.

@ericdill

This comment has been minimized.

Contributor

ericdill commented Sep 25, 2017

Oh have fun 😎 🏖 🍸 !

Sorry for interrupting your vacation 😞

@mingwandroid

This comment has been minimized.

Collaborator

mingwandroid commented Sep 25, 2017

No problem! Hopefully we can get constructor releases sorted out for you soon.

@kalefranz

This comment has been minimized.

Member

kalefranz commented Sep 29, 2017

@mingwandroid

I've been cautioning people not to use noarch: python for a while now. It's just not ready for prime-time I'm afraid

Are there any bugs with conda proper that you're aware of, or is that statement more specific to constructor?

@ericdill ericdill referenced this issue Oct 9, 2017

Merged

Use noarch Python #6

@jakirkham

This comment has been minimized.

Contributor

jakirkham commented Oct 9, 2017

Was thinking about this a few days ago. Would it be possible to use conda or conda-build to convert noarch: python packages into an architecture/Python version specific package before bundling everything together with constructor? If so, that would make conda and/or conda-build a requirement for creating an installer with constructor, but would not require either to actually run the install or use the resulting deployment. Thoughts on this?

@mingwandroid

This comment has been minimized.

Collaborator

mingwandroid commented Oct 9, 2017

@ericdill, I should note that none of the packages in AD5 came from the URL (or rather, from the free channel) you linked to. This is the correct noarch URL: https://repo.continuum.io/pkgs/main/noarch/

@ericdill

This comment has been minimized.

Contributor

ericdill commented Oct 9, 2017

@mingwandroid -- thanks for the corrected link. What's the relationship between pro, free and main?

@jakirkham -- 👍

@mingwandroid

This comment has been minimized.

Collaborator

mingwandroid commented Oct 9, 2017

pro is something I've never had much dealings with, I think it's mostly deprecated. free contains packages pre-dating AD5. main is the new stuff. Nearly all of the packages are based on forks of conda-forge recipes and all C and C++ packages are compiled with the brand new pseudo cross-compilers on macOS and Linux.

@ericdill

This comment has been minimized.

Contributor

ericdill commented Oct 31, 2017

@mingwandroid @kalefranz any updates on this?

@jakirkham

This comment has been minimized.

Contributor

jakirkham commented Nov 11, 2017

Is this still an issue with the latest constructor? I actually don't know the answer. Just am curious if people have tried and/or if there is a fix?

@mingwandroid

This comment has been minimized.

Collaborator

mingwandroid commented Nov 12, 2017

Nothing to add from me, sorry. I do not like the idea of converting from noarch to arch though. You end up creating two different versions of the same thing and that seems like bad technical debt.

@kalefranz

This comment has been minimized.

Member

kalefranz commented Nov 13, 2017

From my comment above

using noarch: python packages in constructor has a lot of implications, including making conda a hard-inclusion in every constructor installer, just like the python interpreter is included in every constructor installer (even when what you're building the installer for doesn't necessarily need python).

That still summarizes my current opinion on the issue.

@CJ-Wright

This comment has been minimized.

CJ-Wright commented Dec 13, 2017

Has there been any progress on this front? noarch seems to be encouraged for conda-forge packaging so it would be very helpful to have this fixed.

@mbargull

This comment has been minimized.

Member

mbargull commented Jan 4, 2018

For the long term, I think including conda (or a subset of it) would be the most consistent choice, but FYI, for the interim: @jakirkham turned his comment about converting noarch packages into an issue over at conda/conda-build#2611.

@jakirkham

This comment has been minimized.

Contributor

jakirkham commented Jan 4, 2018

While that definitely could turn into something used by constructor, there is still value in being able to convert between different types of packages regardless of the intent. One can already do this for the arch packages as long as they are truly platform independent. It's only natural to extend it to the noarch: generic and noarch: python cases.

@mingwandroid

This comment has been minimized.

Collaborator

mingwandroid commented Jan 10, 2018

This is something that the community will probably have to address in its entirety of it is to be fixed.

I personally will not have time and AD can and does work around it by not using any noarch: python packages.

Even if it were fixed I'd still be very hesitant to make use of it due to permissions issues around .pyc files in multi-user setups.

@mbargull

This comment has been minimized.

Member

mbargull commented Jan 11, 2018

@mingwandroid: How do those .pyc permission issues occur; are they tracked somewhere over at https://github.com/conda/conda/issues?

@kalefranz

This comment has been minimized.

Member

kalefranz commented Jan 12, 2018

@mbargull I think the statement above from @mingwandroid

I've been cautioning people not to use noarch: python for a while now. It's just not ready for prime-time I'm afraid and this is a good example of that (others include permissions with .pyc files, menu shortcuts .. the list probably goes on and on).

Was specifically targeted at constructor, and not necessarily conda. If there are outstanding issues in conda for noarch, I'm not aware of them.

cfobel added a commit to sci-bots/microdrop that referenced this issue Jan 16, 2018

feat(installer): move noarch pkgs to site-packages
Explicitly move noarch packages into `Lib/site-packages` as a workaround
to [this issue][i86] with lack of `constructor` support for `noarch`
packages.

[i86]: conda/constructor#86 (comment)
@cfobel

This comment has been minimized.

cfobel commented Jan 19, 2018

For what it's worth, I've had luck working around the issue on Windows by including the following in a post-install.bat script:

@echo off
REM Explicitly move noarch packages into `Lib/site-packages` as a workaround to
REM [this issue][i86] with lack of `constructor` support for `noarch` packages.
REM
REM [i86]: https://github.com/conda/constructor/issues/86#issuecomment-330863531
IF EXIST site-packages (
REM Move directory modules.
for /D %%i in (site-packages/*) do IF "%%~xi" == "" (IF NOT EXIST Lib\site-packages\%%i (
    echo Move noarch package: %%i
    move site-packages\%%i Lib\site-packages
))
REM Move file modules.
for %%i in (site-packages/*.py) do IF "%%~xi" == ".py" (IF NOT EXIST Lib\site-packages\%%i (
    echo Move noarch package: %%i
    move site-packages\%%i Lib\site-packages
))
rmdir /S/Q site-packages
)

It moves the packages from the site-packages directory in the root directory into the proper Lib/site-packages location. Since the packages are in the installed index, conda seems to be happy and the packages can be uninstalled, etc. as normal.

Maybe this hack will be of use to someone else until the common conda static executable proposed by @mingwandroid is completed.

cfobel added a commit to sci-bots/microdrop that referenced this issue Jan 19, 2018

fix(installer): also move file noarch packages
Prior to this commit, only noarch packages containing directory modules
were explicitly moved into `Lib/site-packages` as a workaround to
[this issue][i86] (lack of `constructor` support for `noarch` packages).

As of this commit, noarch packages containing file modules (i.e., `.py`
files) are also moved.

[i86]: conda/constructor#86 (comment)

@jakirkham jakirkham referenced this issue Jan 20, 2018

Merged

REL: 0.2.1 #2

@jschueller jschueller referenced this issue Feb 12, 2018

Merged

Rearch #17

Chilipp added a commit to Chilipp/psyplot-conda that referenced this issue May 13, 2018

Move noarch packages in post-install script
This commit uses the fix in conda/constructor#86 (comment) to move noarch packages to the site-packages directory

yuvipanda added a commit to jupyterhub/the-littlest-jupyterhub that referenced this issue Jun 27, 2018

Use miniconda3 directly, do not use conda constructor
Conda constructor wasn't providing enough use to justify its
restrictions:

1. It doesn't work with most of conda-forge due to
   conda/constructor#86 (comment).
   Almost all the packages we want to ship are in conda-forge.
2. There is no way to pass environment variables to the post-install
   script, which makes customization super hard.
3. Can't bundle pip packages, so we need internet access to install
   packages anyway. This negates the 'offline installable' aspect of
   conda constructor.
4. We need an installer script that users can curl into bash regardless.

Downloading & installing miniconda straight up seems simple enough
for now, and has a reasonable upgrade pathway. I think it'll work out ok!

Chilipp added a commit to ARVE-Research/gwgen that referenced this issue Jul 1, 2018

Apply post install patch for noarch packages
This commit uses the fix in conda/constructor#86 (comment) to move noarch packages to the site-packages directory

ericpre added a commit to ericpre/constructor that referenced this issue Aug 1, 2018

Blacklist noarch python packages
Constructor is not yet compatible with noarch python packages as per the
discussion in conda#86

This "fix" allows us to continue using constructor like we have been
previously before noarch: python packages started appearing. When the
fixes described in conda#86 land this fix can be reverted.

ericpre added a commit to ericpre/constructor that referenced this issue Aug 1, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment