Make some setup.py paths exclude-able

[smontanaro]

BPO	8406
Nosy	@ronaldoussoren, @ned-deily, @merwok, @bitdancer, @willingc, @Carreau
Files	setup.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ronaldoussoren'
closed_at = <Date 2016-06-23.04:51:11.577>
created_at = <Date 2010-04-15.03:12:31.715>
labels = ['OS-mac', 'type-bug', 'build']
title = 'Make some setup.py paths exclude-able'
updated_at = <Date 2016-07-22.16:21:35.335>
user = 'https://github.com/smontanaro'

bugs.python.org fields:

activity = <Date 2016-07-22.16:21:35.335>
actor = 'mbussonn'
assignee = 'ronaldoussoren'
closed = True
closed_date = <Date 2016-06-23.04:51:11.577>
closer = 'willingc'
components = ['Build', 'macOS']
creation = <Date 2010-04-15.03:12:31.715>
creator = 'skip.montanaro'
dependencies = []
files = ['16930']
hgrepos = []
issue_num = 8406
keywords = ['patch']
message_count = 22.0
messages = ['103181', '103189', '103192', '103193', '103203', '103206', '103221', '103248', '103250', '103254', '269103', '269170', '270873', '270877', '270879', '270881', '270882', '270883', '270966', '270987', '270988', '270996']
nosy_count = 7.0
nosy_names = ['ronaldoussoren', 'ned.deily', 'eric.araujo', 'Arfrever', 'r.david.murray', 'willingc', 'mbussonn']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue8406'
versions = ['Python 3.2']

[smontanaro]

The topic of the vileness of Fink or MacPorts came up in python-dev when
discussing building a Mac installer. I remembered that a couple /sw and
/opt/local directories are searched unconditionally, making it a bit
more challenging for someone to create a framework build which didn't
rely on one or the other.

The attached makes the addition of /sw/... or /opt/local/... directory
searches conditional on the presence of /sw/bin or /opt/local/bin in
PATH. I don't have more time to expand upon the concept more, but it
might also be useful to exclude other platform-dependent optional
directories from searching (such as /opt/sfw for Solaris).

[ned-deily]

I think this is essentially a duplicate of bpo-7713.
(Also there doesn't seem to be an "attached".)

[smontanaro]

Let's try this again.

[smontanaro]

The patches referenced in bpo-7713 just unconditionally delete certain
directories from the search path(s). This patch (now attached) allows
the user to control that. I presume, for instance, if the MacPorts
folks don't want /usr/X11/... in search paths that they don't have
/usr/X11/bin in PATH when they build Python. This concept can be
extended to other such directories. I was mostly concerned about the
Fink and MacPorts directories since they came up in the thread on
python-dev.

[ned-deily]

Without testing it, I'd guess the patch wouldn't hurt but it also wouldn't affect the OS X installer build as the installer script builds a local copy of Sleepycat db4 (since none is included in OS X) and supplies it as an additional local library/include file which is always found first by the setup.py code, so it never gets to searching the additional paths affected by the patch. If you are building manually and don't supply a local lib, then the build of the dbm extension will either fail or it will try to use a MacPorts or Fink version if available.

[ronaldoussoren]

Ned: the patch does have a (positive) effect when building the OSX installer: the build-installer script creates private builds of a number of libraries, but setup.py can detect other libraries (such as libintl) as well and we don't guard against that at the moment.

Skip: I'm +0 on including this functionality, but haven't looked at the patch yet.

To be honest I'd must rather remove /opt/local and /sw entirely from setup.py and add new configure flags to specify the location of a number of libraries (e.g. 'configure --with-sleepycat-db=/opt/local'). Explicit configure flags is what other OSS packages seem to use to specify that you like to use a library that is not in /usr or /usr/local.

[smontanaro]

Ronald> To be honest I'd must rather remove /opt/local and /sw entirely
Ronald> from setup.py and add new configure flags to specify the
Ronald> location of a number of libraries (e.g. 'configure
Ronald> --with-sleepycat-db=/opt/local'). Explicit configure flags is
Ronald> what other OSS packages seem to use to specify that you like to
Ronald> use a library that is not in /usr or /usr/local.

Right, but then you still have the problem with /usr/local (note the package
patches mentioned in the other issue), and you'd have to go through that
dance for every bit of functionality which might be found in an external
library.

Someone else (David Cornapeau?) mentioned he just uses a chroot jail to
build numpy and scipy installers. That might ultimately be the best way to
do things.

Skip

[ned-deily]

Ronald, as far as I can see, the patch as it stands only addresses dbm so more work is needed to address all the other cases, the ones that actually matter. bpo-7713 describes a more serious, general problem because a number of other builders / distributers have the same problem, and not only on OS X. Everyone currently needs to maintain their own set of patches on top of every release to modify setup.py's search behavior. That's where the effort should go, I think.

[ned-deily]

Skip, I saw David's chroot remark but I'm having a hard time envisioning how one could use that as a practical matter to build an OS X framework without some *serious* changes to the current configure(s), Makefile(s), and installer script.

[ned-deily]

Sorry, Ronald, of course that's what you were suggesting with the proposed configure --with options. +1 on that. I'd just want to ensure that any solution addresses any needs in bpo-7713, too. Something like that should. There are probably other open issues in a similar vein.

[willingc]

This issue is languishing. Recommend closing the issue.

[bitdancer]

We don't close issues just because they are languishing. We'd need to know that the issue has been superseded or is otherwise no longer an issue. It sounds like this one falls into that category, since another issue is noted as a possible superseder, but I don't know enough about the issues involved to be sure.

[ronaldoussoren]

I don't understand why this issue was closed. Apparently it was closed because it is an old issue without checking if the issue is still relevant???

To be honest the way setup.py looks for libraries could use some attention, the current methods is a bit too magical to my taste. It is also too easy to accidentally depend on libraries you don't want to rely on (I have to create CPython builds for internal redistribution on a separate VM to avoid relying on HomeBrew).

P.S. Sorry about the late reaction, the bug tracker mails about this came in just before I went on a trip. I'm finally catching up with older mail.

[willingc]

Ronald, while I may have closed this issue prematurely in other folks' view, I did not do so without thought or reflecting if the issue was relevant.

My recommendation to close the issue was based on several factors:

• the desire to assist triaging issues that are not actionable in their current state

• the lack of activity for over 6 years on an overly broad issue topic

• an opportunity for a more suitable issue to be opened which more accurately reflects the development needed and actions needed

• this issue's history, if necessary, could easily be referred to in a better scoped issue

Overall, I see closing this issue as an opportunity to address in a new issue the specific setup.py improvements that would be helpful today.

[ronaldoussoren]

Did you create that new issue? If not this feature request will likely be forgotten.

Hmmm... bpo-7713 appears to be good enough for remembering that there is a problem with the way setup.py looks for libraries.

[bitdancer]

Right. Only close an issue if there is an existing open issue that addresses the topic better. Age alone is never a criteria for closing issues.

Since Ronald agrees the other issue is enough we'll leave this closed, but it wouldn't be crazy to reopen it either.

[bitdancer]

Heh, that should have been "only close an old issue that hasn't been otherwise resolved if...".

[willingc]

Fair enough.

Ronald and David, I respect your suggestions and thoughts, and I will not triage or close old issues. FWIW, I was trying to be helpful. We regularly close issues that are 1+ year old on Jupyter/IPython or rescope the issue to be actionable for contribution.

[Carreau]

We regularly close issues that are 1+ year old on Jupyter/IPython or rescope the issue to be actionable for contribution.

Sorry if me doing that on IPython/Jupyter recently have put you (willingc) in a bad position in here, sincere apologies, you are doing a great job.
On the Jupyter/IPython repositories we find that too many open issues and/or patch proposals open tend to repel contributors away. So we try to actively clean things up, especially the ones that are old and inactive. It's true that project might have different policies though.

I, personally, would prefer some of my patch/issues to be closed with a "Sorry, we don't have time to review that and that's going to be controversial anyway, though we appreciate you doing the effort" than leave them in limbo with no responses.

It happen of course that committers overzealously close issue, of course – we are human after all and make mistakes, whether we have years of experience, or a new contributor who does not know the protocol.  It is always a good time to show the example, and show how we could have improved the process.I don't think that making a mistake should be seen as dramatic as it's often communicated. After all, we have revision control, and we can reopen right ?

For example, I think that recently we [IPython/Jupyter] did a better job at reopening and explaining what could have been done better – and actually do it – instead of what was not done. But most of the time the closing of languishing issue was the right choice, and the cost/benefit, despite the false positive was worth it.

I also think the closing an issue [with or without patch], make the author react if they really do care, often more than that if you just ask for a status update. This author, in general, is often more aware of potential new issues that supersede the closed one and thus can link to it.

Again, I know that working on multiple projects can be hard and that practices that differ between project can also be hard to follow, and my recent spring-cleaning might have biased your practices here. Thanks for your hard work both on IPython/Jupyter, here and at on the PSF board, and sorry if it put you into trouble.

[bitdancer]

No, no trouble. We appreciate Carol's efforts! As you say, any changes can be reversed; we prefer that people exercise triage privileges and make mistakes than that they feel too cautious to do anything. If our tone came across as confrontational that was unintentional. It is, rather, a matter, as you say, of communicating our community's style and standards. The only time a problem arises is if someone ignores or flat out rejects the feedback instead of accepting the community conventions or engaging in civil discussion about them, and that is certainly not the case here!

There is also the possibility that our approach is not optimal, but that is a meta discussion that would take place on python-dev.

[ronaldoussoren]

I'm sorry if my comment was too harsh, it wasn't intended as such. I was surprised at seeing a number of e-mails about issues that were closed because they were out of date.

I don't agree with closing old issues when there is actionable content in them because that might result in losing valuable information as closed issues are effectively gone.

Closing old issues is fine when it is clear that the are no longer relevant, but that should IMHO be done with a message that explains why the issue is closed (such as when an issue can no longer be reproduced in supported branches on Python and/or on supported OS releases).

BTW. Feel free to ping me about issues assigned to me when they appear to be no longer relevant but you're not sure.

[Carreau]

[2 nice response]

Thanks for the nice responses, we all know it can be hard to find the right balance between making a concise comment, and being delicate enough.

Thanks all for your hard work on CPython, and looking forward to have time to contribute more !