Refactor requirements file parsing

[qwcode]

Two major changes:

1. Re-use the optparse options in pip.cmdoptions and build a standard optparse parser, instead of maintaining a custom parser. This improves consistency and reduces bugs when we make options changes, and should make the eventual click-rehaul of our cli easier.
2. As a result of No instructions for how to install pip #1, the call stack is simpler:
   • from: parse_requirements -> parse_content -> parse_line
   • to: parse_requirements -> process_line

Changes beyond 1 & 2:

• Adjusting the tests to match
• Adding more tests (specifically testing every supported option, and functional test that confirms Add --install-options and --global-options to the requirements file parser. #2537)
• Minor cosmetics

There are some behavior changes:

• The parsing is now consistent with the cli parsing. For example, before we were allowing things like "-i=url", and possibly other subtle differences.
• We now allow multiple of the same option (but not different options) on the same line, largely due to not having an easy way to detect this using optparse. Is there a way?

[qwcode]

@pfmoore thoughts? since you looked at this code recently?

[pfmoore]

I've had a quick look. I don't have much time over the next few days to do a full review, but the new code looks a lot clearer than the previous version, which is great. Processing looks fine.

There's a lot of changes to the tests, which I haven't looked at in detail but which seem to be mostly because they are testing internal functions of the old code, and so are no longer applicable. I'm not sure to what extent (either before or after this patch) we test the "higher level" aspects of requirement file processing (i.e. for various requirement files, does the parsing give the overall results that are expected), but that's not directly relevant to this patch - it may be useful to review as part of the more general "improve our tests" process. But for this patch, I'm assuming (for now) that test coverage remains sufficient.

I'll try to get time to do a more thorough review, but it might not be for a couple of weeks, so don't wait on me. Overall, though, the change looks good to me.

[qwcode]

thanks @pfmoore . I think the tests are overall improved in this PR. Specifically, the functional test now confirms the override behavior for real using "--prefix", i.e. it confirms a prefix value in the req file overrides the value in the cli. Also, there are now tests for every supported option.

thoughts @gvalkov ?

[gvalkov]

Wouldn't this prevent requirement lines like the following from working?

req1 >= 1.0   # args = ['req', '>=', '1.0']

I think all positional arguments should be consumed first (i.e the requirement specifier) and then only the options are to be passed to parse_args(). Something along the lines of:

if ' --' in line:
    req, args = line.split(' --', 1)
    args = shlex.split('--%s' % args)
    opts, args = parser.parse_args(args)
else:
    req = line
    opts = None

if opts:
   ...

[qwcode]

good catch. I think I'll remove the ">1" check all together, and just pass all the args (as a joined string) into the InstallRequirement constructor. If they did really try to put multiple requirements on one line, it will fail at that point. that's good enough I think.

and I'll certainly add a test for this.

[gvalkov]

That can work, yes. Hopefully nobody figures out that arguments and options can be interspersed :)

pillow --install-option one >= --global-option=two 2.8.1

[qwcode]

I'm conflicted here, but I guess I'm biased towards keeping it simple, and letting this possibility exist.

the custom parsing could get more complicated if we ever support short options, since we'd be dealing with "--" and "-"

[pfmoore]

Agreed - keep it simple. If this does become a genuine issue, I'd suggest something more along the lines of

    opts, args = parser.parse_args(orig_args)
    if len(args) > 1:
        positions = [orig_args.index(a) for a in args]
        if (the values in positions aren't all next to each other):
            raise an error

But that's a costly check, so I wouldn't bother unless it's a genuine issue.

[gvalkov]

I really like this - it's definitely an improvement/simplification over the previous implementation.

Using spaces in the requirement specifier is allowed, right? I think I use it like that all the time (i.e pillow >= 2.8.1). This PR would mandate that all specifiers single words (i.e pillow>=2.8.1). It's surprising that there wasn't a test to catch this.

Happy to hear that pip is considering click for its command-line interface - I find the sub-parser handling in it nicely done, but I'm not too excited about functions with up to 20 arguments (as would be the case for the install command).

[qwcode]

Using spaces in the requirement specifier is allowed, right?

yes, the only caveat is the need to quote when using as cli arguments in the shell