pythonw should not redefine PYTHONEXECUTABLE #446
Comments
|
On Wed, Sep 16, 2015 at 8:09 AM, Thomas Robitaille <notifications@github.com
but maybe it should be set to the executable in the app bundle, as that's
indeed -- it would be nice to have that all "just work" that way it has in Does anyone know if there is a technical reason it's not done the way it is -Chris Christopher Barker, Ph.D. Emergency Response Division |
|
|
|
Why do you want to use |
|
"""PYTHONEXECUTABLE is set to fix argv[0] in Modules/main.c""" Is there a reason that that shouldn't be the one inside the app bundle? |
|
OR si what the needs is a In any case, still nicer to get rid of the need for pythonw -- easier cross platform coding. |
|
@ilanschnell @ChrisBarker-NOAA Here is a nice example for why from PyQt4.QtGui import QApplication, QMainWindow, QMenu
app = QApplication([''])
main = QMainWindow()
mbar = main.menuBar()
menu = QMenu(mbar)
menu.setTitle("&File")
mbar.addMenu(menu)
main.show()
main.raise_()
app.exec_()What you will see is that the menu doesn't appear straight away (you have to switch application then back to get it to work). Now try running the program again with Basically Now let's take a more complicated example - let's say I want to install the Now if I try and start glue on MacOS X 10.9 and 10.10 by running the So it uses the normal Python, and is the same as running the simple script higher up with Nothing changes because of the export statement in the and so when I run The reason why people usually never install the GUI packages with Does this make more sense? |
|
I am cc-ing @ChrisBeaumont in case I missed anything out |
|
That sounds right to me. It would be great if |
|
@ilanschnell - I'm now seeing issues due to this in the latest version of MacOS X. If I check out the source code of the glueviz package (https://github.com/glue-viz/glue) and then do: then when I start up glueviz with it still doesn't work, and still creates a For comparison, if I install the conda I think this is still the same issue as here - can you suggest a workaround? |
|
@ilanschnell - if I install with: then change the first line of to it then works fine. Is there any way to make it so that when installing with: the first line of the script is set correctly? |
|
On Sun, May 22, 2016 at 10:00 AM, Thomas Robitaille <
/Users/tom/miniconda3/envs/dev35/bin/pythonw which is the python convention for windowing apps. it then works fine. Is there any way to make it so that when installing
Alternatively, you could say Anaconda is broken, as the python.org build This was posted as an issue on the setuptools project a while back: I suppose one of us needs to submit an actual patch.. -CHB
Christopher Barker, Ph.D. Emergency Response Division |
|
From @ChrisBarker-NOAA on Dec 12 2016, Anaconda ML: conda folks: PING!!! one more good reason to make your default python interpreter GUI compatible -- like the python.org build does -- it would solve all these problems. As for getting ipython to run under pythonw, I have no idea :-( unfortunately the setuptools entry-points does not do the right thing -- though I"m not sure that ipython uses that anyway. That part may be a question for the ipython folks. -CHB |
|
+1 fo solving this! conda convert does not work for scripts with entry points if they use wxpython for example. |
|
Also seems to affect Ginga (see ejeschke/ginga#441). |
|
Since this is getting Pinged again, I"ll try a little summary: Apple, in its Infinite Wisdom doesn't allow regular old command line apps to access the GUI (or at least not properly). GUI apps MUST be run from inside an "application bundle" -- you can look elsewhere for details, but that's how I understand it. So regular old unix-style-build provided python executable won't work. History: back in the day, the windows build of python provided a "python" and "pythonw" executable (I don't know if it still does...) and one would bring up a console for you (for stdout and stdin), and one wouldn't. Meanwhile, the Mac build did a similar thing -- so it was kind of a standard that GUI apps would be run with "pythonw" and console apps with "python". Eventually, the folks creating the Mac builds developed a (kinda kludgy) way to make a small executable that would run inside an app bundle, but also be lightweight enough to be the main command line executable as well -- so for quite some time, the python.org builds shipped a "python" and a "pythonw", but they were aliases for the same thing, and both would behave exactly the same. However, there was also a bunch of other complications and Mac-specific stuff in the python.org builds: "Framework Builds", "Fat binaries", PPC vs Intel vs 32bit vs 64bit.... So someone hacked together a build system that would build various combinations of these builds. But there is the rub -- there is no easy way to build a regular old unix style (non-framework) build with the pythonw executable. So when Continuum came along to setup Anaconda, they chose the Unix-style build, as they didn't want all the Framework and Fat build overhead. But this did not get them the nifty pythonw hack. So, in order to make GUI apps work, continuum provides the "python.app" package, which provides a application bundle with a python executable in it, (and a couple other things to make qt work) and a "pythonw" shell script in bin that re-directs to that app bundle python (but setting PYTHONEXECUTABLE to the regular executable) -- this allows the OS to see an executable in an application bundle, so ti's happy, but the rest of Python sees the regular one, which it needs to, as the rest of the Python install is not in the bundle. So what's the problem? I've lost track of all the issues, but I think that the fact that pythonw is a shell script, rather than a proper executable causes some issues. But I know that the main issue is that setuptools doesn't have respect the "pythonw" concept. So if setuptools is used to install a script with "entry points" or whatever, it used plain "python", so the script won't work. Also, since pythonw is a shell script that re-directs, you can't just hack the startup script that setuptools builds, and you can't run "setup.py install" with pythonw either. (and no, I don't fully understand these complications, but I do know the simple and obvious things don't work) Solutions: I think there are essentialy two possible routes to a solution:
Continuum hasn't addressed this in ages (I think GUI apps are just not a major driver, and the pythonw solution they have mostly works). My suggestion, if someone can find the energy and time to work on this, is to contribute the fix to the conda-forge python build. Then folks would have access to it, and maybe Continuum would pick it up for the official builds once it proves itself. https://github.com/conda-forge/python-feedstock Anyway, it'll never get fixed unless someone puts some time into it! |
|
Commands line apps have access to the GUI. The facilities that a plist
endows an application with can be gained programmatically too for the most
part (icons, file associations and some permissions).
Please check out, for example the rstudio binary. It is just a normal
executable.
…On Mar 24, 2017 3:46 PM, "Chris Barker" ***@***.***> wrote:
Since this is getting Pinged again, I"ll try a little summary:
Apple, in its Infinite Wisdom doesn't allow regular old command line apps
to access the GUI (or at least not properly). GUI apps MUST be run from
inside an "application bundle" -- you can look elsewhere for details, but
that's how I understand it. So regular old unix-style-build provided python
executable won't work.
History: back in the day, the windows build of python provided a "python"
and "pythonw" executable (I don't know if it still does...) and one would
bring up a console for you (for stdout and stdin), and one wouldn't.
Meanwhile, the Mac build did a similar thing -- so it was kind of a
standard that GUI apps would be run with "pythonw" and console apps with
"python".
Eventually, the folks creating the Mac builds developed a (kinda kludgy)
way to make a small executable that would run inside an app bundle, but
also be lightweight enough to be the main command line executable as well
-- so for quite some time, the python.org builds shipped a "python" and a
"pythonw", but they were aliases for the same thing, and both would behave
exactly the same.
However, there was also a bunch of other complications and Mac-specific
stuff in the python.org builds: "Framework Builds", "Fat binaries", PPC
vs Intel vs 32bit vs 64bit.... So someone hacked together a build system
that would build various combinations of these builds. But there is the rub
-- there is no easy way to build a regular old unix style (non-framework)
build with the pythonw executable.
So when Continuum came along to setup Anaconda, they chose the Unix-style
build, as they didn't want all the Framework and Fat build overhead. But
this did not get them the nifty pythonw hack.
So, in order to make GUI apps work, continuum provides the "python.app"
package, which provides a application bundle with a python executable in
it, (and a couple other things to make qt work) and a "pythonw" shell
script in bin that re-directs to that app bundle python (but setting
PYTHONEXECUTABLE to the regular executable) -- this allows the OS to see an
executable in an application bundle, so ti's happy, but the rest of Python
sees the regular one, which it needs to, as the rest of the Python install
is not in the bundle.
So what's the problem?
I've lost track of all the issues, but I think that the fact that pythonw
is a shell script, rather than a proper executable causes some issues. But
I know that the main issue is that setuptools doesn't have respect the
"pythonw" concept. So if setuptools is used to install a script with "entry
points" or whatever, it used plain "python", so the script won't work.
Also, since pythonw is a shell script that re-directs, you can't just hack
the startup script that setuptools builds, and you can't run "setup.py
install" with pythonw either. (and no, I don't fully understand these
complications, but I do know the simple and obvious things don't work)
Solutions:
I think there are essentialy two possible routes to a solution:
1. patch setuptools to use pythonw for windowed applications.
- However, there are issues on the setuptools project, and no one
seems interested in doing that. plus setuptools is a bit of a
Frankenstein's monster with an uncertain future (it's not going away, but
it may be substantially refactored). And it's not as simple as replacing
"python" with "pythonw", due to the redirection script. But if someone
wants to work on a patch and submit it -- maybe it will take.
1. update the Anaconda python build to build the python/pythonw
executable. I think the is the easiest and most robust way to solve this
problem. It would require some hacking of the build scripts for python, so
someone that knows a bit about autotools would have to do it. I planned to
sit down with Ned Deily last year at the PyCon Sprints to figure it out,
but ended up working on something else :-(.
Continuum hasn't addressed this in ages (I think GUI apps are just not a
major driver, and the pythonw solution they have mostly works). My
suggestion, if someone can find the energy and time to work on this, is to
contribute the fix to the conda-forge python build. Then folks would have
access to it, and maybe Continuum would pick it up for the official builds
once it proves itself.
https://github.com/conda-forge/python-feedstock
Anyway, it'll never get fixed unless someone puts some time into it!
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#446 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AA_pdNwFwdXfOvDebrIbCZFdwqwgjnydks5ro-VNgaJpZM4F-eDF>
.
|
|
On Fri, Mar 24, 2017 at 8:53 AM, Ray Donnelly ***@***.***> wrote:
Commands line apps have access to the GUI. The facilities that a plist
endows an application with can be gained programmatically too for the most
part (icons, file associations and some permissions).
If you know how to do that, then please enlighten us -- I honestly do NOT
understand the details, but I know know that Python has been hacking around
this for years (maybe that's legacy) and that, or course, the regular old
python executable does not work.
Please check out, for example the rstudio binary. It is just a normal
executable.
Are you sure? the python executable in the python.org builds looks like an
ordinary binary, but it does some re-direction trickery under the hood.
But if you could figure out what rstudio is doing, maybe we can use that.
(I'm going to guess that it is either doing a re-direction trick ala the
python.org build, or it is doing, as you say, "The facilities that a plist
endows an application with can be gained programmatically" -- but we don't
want the regular python executable to do that endowing in all cases.
Also -- this is not about "The facilities that a plist endows an
application with" -- it is about the initial startup and access to a Window
manager.
…-CHB
|
|
On Fri, Mar 24, 2017 at 4:15 PM, Chris Barker <notifications@github.com>
wrote:
On Fri, Mar 24, 2017 at 8:53 AM, Ray Donnelly ***@***.***>
wrote:
> Commands line apps have access to the GUI. The facilities that a plist
> endows an application with can be gained programmatically too for the
most
> part (icons, file associations and some permissions).
>
If you know how to do that, then please enlighten us -- I honestly do NOT
understand the details, but I know know that Python has been hacking around
this for years (maybe that's legacy) and that, or course, the regular old
python executable does not work.
Please check out, for example the rstudio binary. It is just a normal
> executable.
>
Are you sure? the python executable in the python.org builds looks like an
ordinary binary, but it does some re-direction trickery under the hood.
Yes, rstudio is a 'normal' a macho executable. It links to the Cocoa
framework and that is what gets you access to the Window manager (Aqua). My
guess regarding Python is that they have legacy stuff in there because of
Carbon support, but I'm not really familiar with the macOS Python
executable internals.
But if you could figure out what rstudio is doing, maybe we can use that.
For RStudio, I made an app bundle which contains nothing more than a
symlink to the real executable and an icon. I did this so that you can pin
an icon to the Dock. I think we should do something like this for all of
our GUI applications and we have some vague plans about adding support to
'menuinst' for this.
(I'm going to guess that it is either doing a re-direction trick ala the
python.org build, or it is doing, as you say, "The facilities that a plist
endows an application with can be gained programmatically" -- but we don't
want the regular python executable to do that endowing in all cases.
Also -- this is not about "The facilities that a plist endows an
application with" -- it is about the initial startup and access to a Window
manager.
I am not sure what you mean by this.
…
-CHB
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#446 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AA_pdNKYFWrubSvLN6hhTJGBBvJki9TCks5ro-w-gaJpZM4F-eDF>
.
|
|
Yes, rstudio is a 'normal' a macho executable. It links to the Cocoa
framework and that is what gets you access to the Window manager (Aqua).
That makes sense -- but we certainly don't want "python" to link to Cocoa.
My
guess regarding Python is that they have legacy stuff in there because of
Carbon support,
Maybe -- but the key here is that the python executable isn't linked to ANY
GUI framework, nor do we want it to be.
But if you could figure out what rstudio is doing, maybe we can use that.
>
For RStudio, I made an app bundle which contains nothing more than a
symlink to the real executable and an icon.
OK, I see now. The thing is, we are trying to solve a different problem
here.
The goal is to have a python executable that can run scripts that use the
GUI.
If we want to make "proper" Mac GUI applications, with an icon, and file
associations, and all that, then you build a App Bundle that contains your
code and the everything else it needs.
That can be done with Py2App or PyInstaller, or ???
And that all works fine.
But if you want o be able to simple run:
python my_script.py
or have setup.py install create a command line app that will run a GUI, you
need to do some of this trickery.
but we don't
> want the regular python executable to do that endowing in all cases.
>
> Also -- this is not about "The facilities that a plist endows an
> application with" -- it is about the initial startup and access to a
Window
> manager.
>
I am not sure what you mean by this.
If you want the plist-y stuff, you make an app bundle for your app, or you
do it at run-time programmatically, as you say. But that's not the problem
at hand. We just want an executable python interpreter that can load up a
GUI lib like QT or wxPython, and be able to access the Window manager.
The main executable can not be linked to Cocoa or anything else GUI.
…-CHB
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov
|
|
@astrofrog, @ChrisBarker-NOAA, @pllim, @ijstokes, @BjornFJohansson: Continuum's Python executable (I tested with 3.6.1) dynamically links to Cocoa's CoreFoundation library (which means our interpreter is GUI compatible). We do not provide an executable that does not do this, whereas CPython releases have two different executables, one which does link to CoreFoundation ( This dynamic linkage can be seen from With respect to entry points, I want I installed the official CPython 3.6.1 and wrote a simple test containing this I would have hoped that installing this via |
|
This is all confusing, but I'm pretty sure you have the issues somewhat confused:
it does indeed link to CoreFoundation: but I don't think that is "Cocoa", but rather other core OS-X stuff: https://developer.apple.com/reference/corefoundation and, indeed, when I try to run, for instance wxPython, with that very executable, I get: but it does run with pythonw. And with the python embedded in the python.app: ~/miniconda3/python.app/Contents/MacOS/python MicroApp.py So -- with the python provided by conda -- you need to run your app with the python inside python.app -- either directly or from pythonw.
This isn't conda build thing -- it's a setuptools thing -- all conda build does is make a package out of whatever your build script installs. Which means we need to:
I think option (1) is the "right" way to solve the problem, but options (3) is the only one you can do right now.
exactly -- setuptools SHOULD use pythonw for gui_scripts -- why do they even HAVE gui_scripts if it's not going to do anything different. But maybe you can hack around it. I tried the really simple approach of hand editing the script that setuptools creates, and replacing "python" with "pythonw" -- that failed, I think because pythonw is a bash script, not a proper executable. But maybe you could try patching in the full path to the python inside the app bundle. or maybe make essentially a copy of the pythonw script. If one of those works, then you at least know what you are trying to hack in.... Once you have it working, we could maybe propose a patch to setuptools (though it it' specific to the conda setup, that's not going to fly). Or maybe conda-build could do some hacking -- there are a number of python-specific features in conda -- why not this one? Another issue -- the conda pythonw script hard-codes the installed path -- you may need to do that, too. I think conda install patches some paths on install, but you'll need to figure out how to trigger that feature... -CHB -CHB |
|
Thanks Chris, Cocoa is not only the GUI part of macOS, it is the entire system that came from NextStep and replaced Carbon. I'll look into this stuff more when I have a chance. |
|
I can never keep track of Apple's Branding... But anyway, we know that simply linking against CoreFramework does not get you the ability to run GUI apps with a Python executable. The folks that understand this stuff can be found on the python-mac list: https://mail.python.org/mailman/listinfo/pythonmac-sig a question there might get you the info yu need. Here is the old discussion: https://mail.python.org/pipermail/pythonmac-sig/2014-August/024079.html Note that reading that again, it seems there may be another trick that could be used -- wx, qt, etc could do what TK does -- not that I have any idea what that is. Though I think I prefer a solution that doesn't require all the GUI frameworks to adapt... -CHB |
|
Hi all, What is the status of this in 2019? I recently packaged a wxpython script as a conda
I am under the impression that this was not solved. Thanks! |
|
I'm pretty sure that the answer to the key question is "no" -- in short. nothing has changed. However, you can add a dependency in your meta.yaml on "python.app" (for only osx)" then you need to make sure that the app gets run with "pythonw", rather than python -- which is pretty easy on a #! line or command line, but I couldn't find a way to use setuptools entry points to respect that :-( I really wish that conda built "pythonw" the same way that the python.org builds do, but I've never had the time to figure out how to do that -- I'm no autoconf expert. |
|
Thanks I will try that. Meanwhile, I found this question at SO. He uses |
|
I found this question
<https://stackoverflow.com/questions/47630802/running-pip-install-with-pythonw-makes-console-scripts-use-pythonw-instead-of-py>
at SO. He uses pythonw pip install ... on Windows in order not to have a
window pop up during the process. If we substitute python setup.py install
in the meta.yaml with python pip install on Mac?
nope -- that's not going to help :-(
…-CHB
|
This is related to (but not exactly the same as) #199, and is also request 907 in the continuum support system.
In Anaconda, unlike other distributions these days, python is different from pythonw, and the latter is needed to run scripts that use e.g. PyQt4 to get everything to work correctly (see also #199)
For some packages, such as the developer version of the glue package, and any other GUI packages, I therefore need to install the package with:
This should in principle ensure that the interpreter used for installed scripts is
pythonw. However, in conda, the pythonw script does the following:before running python.app.
This is a problem, because it means that running
basically has no effect compared to installing with python, and the installed scripts will just be run with python instead of pythonw or python.app (this behavior is unlike other distributions). If I comment out this line, everything works as expected.
Would it be possible to remove the line setting PYTHONEXECUTABLE from
pythonw? At the moment, I cannot install GUI packages properly withpythonw setup.py install, and making a conda package each time before installing is not realistic, since I am doing a lot of development on these packages, so need to install them often.Of course, the ideal solution is that
pythonshould just be the same aspythonwas discussed in #199, but at the very least in the mean time, I think that theexport PYTHONEXECUTABLEline should be removed since it causes issues when installing scripts.I am using the OSX version of anaconda by the way.
(also cc-ing @ChrisBarker-NOAA and @asmeurer since you were discussing this in #199)
The text was updated successfully, but these errors were encountered: