Skip to content

added: --install-option on requirements file #271 #515

Closed
wants to merge 1 commit into from

6 participants

@conrado
conrado commented Apr 23, 2012

Hi,

I'm not really sure this is the best way about it... but it works.

Bye.

@hltbra
Python Packaging Authority member
hltbra commented Apr 27, 2012

Someone asked about it in StackOverflow: http://stackoverflow.com/questions/10278708/pip-pass-custom-options-to-installer-with-requirements-txt

After merging it would be nice to write an answer there.

@bernardofire

@hltbra, I already commented the question in stackoverflow saying about this pull request.

@conrado conrado and 1 other commented on an outdated diff May 1, 2012
pip/req.py
@@ -81,6 +83,15 @@ def from_line(cls, name, comes_from=None):
path = os.path.normpath(os.path.abspath(name))
link = None
+ inst_opt_pos = name.find('--install-options=')
+ # should we use `argparse` or safer method instead?
@conrado
conrado added a note May 1, 2012

If anyone could review this bad comment line and tell me if there's a better way to go about this I can go and fix it. argparse seemed too complicated.

Example requirements.txt and caveats

pyzmq>=2.2.0 --install-options="--zmq=/usr/local/lib"

the string --install-options= is searched for on the line, and chomped. Doesn't do any error checking, we just grab whatever is inside the quotes. There's no escaping double-quotes, or spaces... which is the next split() in L#90 so you can pass multiple parameters.

So, you could have:
--install-options="--libjpeg=/usr/local/lib --libpng=/usr/local/lib"
but
--install-options="--libjpeg=/home/conrado/library\ builds/"
will fail in a very bad way. So, is this acceptable?

@pnasrat
Python Packaging Authority member
pnasrat added a note May 13, 2012

Thanks for the pull request.

I'd say we definitely should have tests around this, and this should be part of your pull request. If you need a hand with that just ask and I'll try help.

So shell parsing and escaping is tricky. For --install-options="--libjpeg=/home/conrado/library\ builds/" does --install-options="--libjpeg=/home/conrado/library\\ builds/" work?

Also have you looked at shlex in the stdlib as this may help with splitting

Also we'd definitely want to support this both in requirements.txt and on the command line, so this is probably the wrong place for the support to be added.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@qwcode
Python Packaging Authority member
qwcode commented May 13, 2012

as for parsing the options from the req, pip is already using optparse internally, and it's supported across all the python versions pip supports

here's a way to realiably parse the req using that.

import optparse
p = optparse.OptionParser()
p.add_option("--install-options")
options, args = p.parse_args(args=['pyzmq>=2.2.0', '--install-options="--zmq=/usr/local/lib"'])

the result:
args = ['pyzmq>=2.2.0']
options.install_options = '"--zmq=/usr/local/lib"'

@pnasrat
Python Packaging Authority member
pnasrat commented May 13, 2012

This is a thought I'm having that I want to preserve in the issue - thinking of other approaches to how to do this, eg we could on the command line using the standard -- to tell pip to stop parsing options and then have a second parse phase to pullout the options for setup.py.

A quick test shows this seems to work on 2.4 and 2.7. I'm not totally sold on that though so

import optparse
parser = optparse.OptionParser()
parser.add_option('--noarg', action="store_true", default=False)
parser.add_option('--witharg', action="store", dest="witharg")
parser.add_option('--witharg2', action="store", dest="witharg2", type="int")
print parser.parse_args([ '--noarg', '--witharg', 'val', '--witharg2=3' ])
(<Values at 0x106018128: {'noarg': True, 'witharg': 'val', 'witharg2': 3}>, [])
print parser.parse_args([ '--noarg', '--witharg', 'val', '--', '--witharg2=3' ])
(<Values at 0x106018248: {'noarg': True, 'witharg': 'val', 'witharg2': None}>, ['--witharg2=3'])
@conrado
conrado commented May 14, 2012

The issue here is that the argument parsing from shell and from the requirements.txt file are "decoupled"(?) So if we change the shell argument options we need to further change it here. I think @pnasrat is addressing this issue with the -- argument end marker suggestion, however that won't really help I think.

Question on functionality, should we have all pip shell options available to requirements.txt? I don't know this, but some options might not be relevant. If we do want to couple the argument parsing on requirements.txt and the shell arguments, we might have to refactor some code and come up with a more elegant pattern than the current solution.

@pnasrat
Python Packaging Authority member
pnasrat commented May 14, 2012

So from my perspective requirements.txt should be able to express everything that is required to install a package. If we need to pass specific options through to setup.py.

@ianb has added another way to do this in #519 although I think you're right we should take a step back and think about what we need to support to fully work. Other cases such as @ niedbalski adding subdirectory support but via the URL #526 point to the need to handle the non trivial cases where just running setup.py won't work.

@jezdez @brosner and @carljm any thoughts?

@qwcode
Python Packaging Authority member
qwcode commented May 14, 2012

yes, reusing the IntallCommand parser somehow and applying it to spec lines in requirements.txt (and overriding the CLI options) would be nice.

@conrado
conrado commented Sep 27, 2012

I have looked into getting more options supported, but it really does need some work on InstallCommand, RequirementSet, possibly parse_requirements.

The way it is now it should work fine as shlex is used to parse the arguments.

@carljm
carljm commented Sep 28, 2012

Hey all - I don't have strong opinions here. Ideally we'd clean up requirements file parsing in general to not be so janky, and be able to offer a general "any option you can use on the command line you can use in a requirements file" guarantee, but that's going to be a sizable chunk of work and I'm not sure it's the highest priority. In practical terms I'm not too concerned about duplicating option names; we shouldn't really be changing them anyway for back-compat reasons.

In this pull request, it seems odd to me to reuse the InstallCommand option parser when really the only thing we are supporting is install_option. I think I'd prefer to go with @qwcode 's suggestion and just use optparse directly to look for the only option we currently support here.

@qwcode
Python Packaging Authority member
qwcode commented Sep 28, 2012

the "reusing the IntallCommand parser" comment was about the possibility of supporting all or most of pip.commands.install options in req files, but not meant to block getting in --install-option support.

@carljm
carljm commented Sep 28, 2012

@qwcode I know, but the current state of the pull request does already reuse the InstallCommand parser, even though none of the options are respected apart from --install-option; that's what I was referring to.

@qwcode
Python Packaging Authority member
qwcode commented Sep 28, 2012

@carljm , ok, I follow now. didn't really analyze how different this new commit was.
I can see the oddness, but otoh, seems harmless and likable.
I can see maybe wanting to add line by line support for -U and --no-deps

more concerned about having tests for it.

@qwcode qwcode commented on the diff Sep 28, 2012
pip/req.py
@@ -113,9 +115,16 @@ def from_line(cls, name, comes_from=None):
url = path_to_url(os.path.normpath(os.path.abspath(link.path)))
else:
+ import shlex
+ line_arguments = shlex.split(name)
+ if len(line_arguments) > 1:
+ from pip.commands.install import InstallCommand
+ parser = InstallCommand().parser
+ (options, args) = parser.parse_args(line_arguments[1:])
+ name = line_arguments[0]
@qwcode
Python Packaging Authority member
qwcode added a note Sep 28, 2012

not following why this is just in the else block? and also not supported for editables.
e.g. we wouldn't be supporting install options for any of these forms

http://stuff.com/my_stash/package.tar.gz
git+http://github.com/frank/frodo@tag#egg=frodo
-e git+http://github.com/frank/frodo@tag#egg=frodo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@conrado
conrado commented Sep 30, 2012

I had a concern when it came to sifting through the RequirementSet and overriding options passed through the command line. What has authority and how should it behave?

In fact, I think I need to look more at the current -r options.

@qwcode
Python Packaging Authority member
qwcode commented Feb 10, 2013

closing this due to #790. will be reviewing that in a few days.

@qwcode qwcode closed this Feb 10, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.