Skip to content
This repository

added: --install-option on requirements file #271 #515

Closed
wants to merge 1 commit into from

6 participants

Conrado Buhrer Hugo Lopes Tavares Bernardo B. Marques Marcus Smith Paul Nasrat Carl Meyer
Conrado Buhrer

Hi,

I'm not really sure this is the best way about it... but it works.

Bye.

Hugo Lopes Tavares
Collaborator

Someone asked about it in StackOverflow: http://stackoverflow.com/questions/10278708/pip-pass-custom-options-to-installer-with-requirements-txt

After merging it would be nice to write an answer there.

Bernardo B. Marques

@hltbra, I already commented the question in stackoverflow saying about this pull request.

pip/req.py
... ...
@@ -81,6 +83,15 @@ def from_line(cls, name, comes_from=None):
81 83
         path = os.path.normpath(os.path.abspath(name))
82 84
         link = None
83 85
 
  86
+        inst_opt_pos = name.find('--install-options=')
  87
+        # should we use `argparse` or safer method instead?
2
Conrado Buhrer
conrado added a note May 01, 2012

If anyone could review this bad comment line and tell me if there's a better way to go about this I can go and fix it. argparse seemed too complicated.

Example requirements.txt and caveats

pyzmq>=2.2.0 --install-options="--zmq=/usr/local/lib"

the string --install-options= is searched for on the line, and chomped. Doesn't do any error checking, we just grab whatever is inside the quotes. There's no escaping double-quotes, or spaces... which is the next split() in L#90 so you can pass multiple parameters.

So, you could have:
--install-options="--libjpeg=/usr/local/lib --libpng=/usr/local/lib"
but
--install-options="--libjpeg=/home/conrado/library\ builds/"
will fail in a very bad way. So, is this acceptable?

Paul Nasrat Collaborator
pnasrat added a note May 13, 2012

Thanks for the pull request.

I'd say we definitely should have tests around this, and this should be part of your pull request. If you need a hand with that just ask and I'll try help.

So shell parsing and escaping is tricky. For --install-options="--libjpeg=/home/conrado/library\ builds/" does --install-options="--libjpeg=/home/conrado/library\\ builds/" work?

Also have you looked at shlex in the stdlib as this may help with splitting

Also we'd definitely want to support this both in requirements.txt and on the command line, so this is probably the wrong place for the support to be added.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Marcus Smith
Owner
qwcode commented May 13, 2012

as for parsing the options from the req, pip is already using optparse internally, and it's supported across all the python versions pip supports

here's a way to realiably parse the req using that.

import optparse
p = optparse.OptionParser()
p.add_option("--install-options")
options, args = p.parse_args(args=['pyzmq>=2.2.0', '--install-options="--zmq=/usr/local/lib"'])

the result:
args = ['pyzmq>=2.2.0']
options.install_options = '"--zmq=/usr/local/lib"'

Paul Nasrat
Collaborator
pnasrat commented May 13, 2012

This is a thought I'm having that I want to preserve in the issue - thinking of other approaches to how to do this, eg we could on the command line using the standard -- to tell pip to stop parsing options and then have a second parse phase to pullout the options for setup.py.

A quick test shows this seems to work on 2.4 and 2.7. I'm not totally sold on that though so

import optparse
parser = optparse.OptionParser()
parser.add_option('--noarg', action="store_true", default=False)
parser.add_option('--witharg', action="store", dest="witharg")
parser.add_option('--witharg2', action="store", dest="witharg2", type="int")
print parser.parse_args([ '--noarg', '--witharg', 'val', '--witharg2=3' ])
(<Values at 0x106018128: {'noarg': True, 'witharg': 'val', 'witharg2': 3}>, [])
print parser.parse_args([ '--noarg', '--witharg', 'val', '--', '--witharg2=3' ])
(<Values at 0x106018248: {'noarg': True, 'witharg': 'val', 'witharg2': None}>, ['--witharg2=3'])
Conrado Buhrer
conrado commented May 14, 2012

The issue here is that the argument parsing from shell and from the requirements.txt file are "decoupled"(?) So if we change the shell argument options we need to further change it here. I think @pnasrat is addressing this issue with the -- argument end marker suggestion, however that won't really help I think.

Question on functionality, should we have all pip shell options available to requirements.txt? I don't know this, but some options might not be relevant. If we do want to couple the argument parsing on requirements.txt and the shell arguments, we might have to refactor some code and come up with a more elegant pattern than the current solution.

Paul Nasrat
Collaborator
pnasrat commented May 14, 2012

So from my perspective requirements.txt should be able to express everything that is required to install a package. If we need to pass specific options through to setup.py.

@ianb has added another way to do this in #519 although I think you're right we should take a step back and think about what we need to support to fully work. Other cases such as @ niedbalski adding subdirectory support but via the URL #526 point to the need to handle the non trivial cases where just running setup.py won't work.

@jezdez @brosner and @carljm any thoughts?

Marcus Smith
Owner
qwcode commented May 14, 2012

yes, reusing the IntallCommand parser somehow and applying it to spec lines in requirements.txt (and overriding the CLI options) would be nice.

Conrado Buhrer

I have looked into getting more options supported, but it really does need some work on InstallCommand, RequirementSet, possibly parse_requirements.

The way it is now it should work fine as shlex is used to parse the arguments.

Carl Meyer
Owner

Hey all - I don't have strong opinions here. Ideally we'd clean up requirements file parsing in general to not be so janky, and be able to offer a general "any option you can use on the command line you can use in a requirements file" guarantee, but that's going to be a sizable chunk of work and I'm not sure it's the highest priority. In practical terms I'm not too concerned about duplicating option names; we shouldn't really be changing them anyway for back-compat reasons.

In this pull request, it seems odd to me to reuse the InstallCommand option parser when really the only thing we are supporting is install_option. I think I'd prefer to go with @qwcode 's suggestion and just use optparse directly to look for the only option we currently support here.

Marcus Smith
Owner

the "reusing the IntallCommand parser" comment was about the possibility of supporting all or most of pip.commands.install options in req files, but not meant to block getting in --install-option support.

Carl Meyer
Owner

@qwcode I know, but the current state of the pull request does already reuse the InstallCommand parser, even though none of the options are respected apart from --install-option; that's what I was referring to.

Marcus Smith
Owner

@carljm , ok, I follow now. didn't really analyze how different this new commit was.
I can see the oddness, but otoh, seems harmless and likable.
I can see maybe wanting to add line by line support for -U and --no-deps

more concerned about having tests for it.

Marcus Smith qwcode commented on the diff September 27, 2012
pip/req.py
@@ -113,9 +115,16 @@ def from_line(cls, name, comes_from=None):
113 115
                 url = path_to_url(os.path.normpath(os.path.abspath(link.path)))
114 116
 
115 117
         else:
  118
+            import shlex
  119
+            line_arguments = shlex.split(name)
  120
+            if len(line_arguments) > 1:
  121
+                from pip.commands.install import InstallCommand
  122
+                parser = InstallCommand().parser
  123
+                (options, args) = parser.parse_args(line_arguments[1:])
  124
+                name = line_arguments[0]
1
Marcus Smith Owner
qwcode added a note September 27, 2012

not following why this is just in the else block? and also not supported for editables.
e.g. we wouldn't be supporting install options for any of these forms

http://stuff.com/my_stash/package.tar.gz
git+http://github.com/frank/frodo@tag#egg=frodo
-e git+http://github.com/frank/frodo@tag#egg=frodo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Conrado Buhrer

I had a concern when it came to sifting through the RequirementSet and overriding options passed through the command line. What has authority and how should it behave?

In fact, I think I need to look more at the current -r options.

Marcus Smith
Owner

closing this due to #790. will be reviewing that in a few days.

Marcus Smith qwcode closed this February 09, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 1 unique commit by 1 author.

Sep 27, 2012
Conrado Buhrer added: --install-option on requirements file #271 1ec6a99
This page is out of date. Refresh to see the latest.

Showing 1 changed file with 13 additions and 2 deletions. Show diff stats Hide diff stats

  1. 15  pip/req.py
15  pip/req.py
@@ -37,7 +37,7 @@
37 37
 class InstallRequirement(object):
38 38
 
39 39
     def __init__(self, req, comes_from, source_dir=None, editable=False,
40  
-                 url=None, as_egg=False, update=True):
  40
+                 url=None, as_egg=False, update=True, options=None):
41 41
         self.extras = ()
42 42
         if isinstance(req, string_types):
43 43
             req = pkg_resources.Requirement.parse(req)
@@ -48,6 +48,7 @@ def __init__(self, req, comes_from, source_dir=None, editable=False,
48 48
         self.editable = editable
49 49
         self.url = url
50 50
         self.as_egg = as_egg
  51
+        self.options = options
51 52
         self._egg_info_path = None
52 53
         # This holds the pkg_resources.Distribution object if this requirement
53 54
         # is already available:
@@ -90,6 +91,7 @@ def from_line(cls, name, comes_from=None):
90 91
         req = None
91 92
         path = os.path.normpath(os.path.abspath(name))
92 93
         link = None
  94
+        options = None
93 95
 
94 96
         if is_url(name):
95 97
             link = Link(name)
@@ -113,9 +115,16 @@ def from_line(cls, name, comes_from=None):
113 115
                 url = path_to_url(os.path.normpath(os.path.abspath(link.path)))
114 116
 
115 117
         else:
  118
+            import shlex
  119
+            line_arguments = shlex.split(name)
  120
+            if len(line_arguments) > 1:
  121
+                from pip.commands.install import InstallCommand
  122
+                parser = InstallCommand().parser
  123
+                (options, args) = parser.parse_args(line_arguments[1:])
  124
+                name = line_arguments[0]
116 125
             req = name
117 126
 
118  
-        return cls(req, comes_from, url=url)
  127
+        return cls(req, comes_from, url=url, options=options)
119 128
 
120 129
     def __str__(self):
121 130
         if self.req:
@@ -558,6 +567,8 @@ def _clean_zip_name(self, name, prefix):
558 567
         return name
559 568
 
560 569
     def install(self, install_options, global_options=()):
  570
+        if self.options and self.options.install_options:
  571
+            install_options += self.options.install_options
561 572
         if self.editable:
562 573
             self.install_editable(install_options, global_options)
563 574
             return
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.