Ignore --find-links lines in requirement checks

Fixes bug #1194807

In quantum, we currently have a URL based requirement:

  http://.../oslo.config-1.2.0a2.tar.gz#egg=oslo.config-1.2.0a2

The requirements check currently ignores this.

It turns out that pip has a bug which doesn't where you can end up with
the oslo.config 1.1.1 code installed. This is because oslo.config>=1.1.0
gets pulled in as a transitive dep and pip gets confused. You can
reproduce with e.g.

  $> pip install \
       http://.../oslo.config-1.2.0a3.tar.gz#egg=oslo.config-1.2.0a3 \
       python-keystoneclient
  $> pip freeze | grep oslo.config
  oslo.config-1.2.0a3
  $> python -c 'from oslo.config.cfg import DeprecatedOpt'
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  ImportError: cannot import name DeprecatedOpt

This is because of a bug with pip where it sees oslo.config-1.2.0a3 and
oslo.config as two unrelated things. It should strip the version part of
the egg= fragment before using it as a package name, but it doesn't.

However, we can simply use the -f/--find-links pip option in our
requirements.txt to add the tarball URL to the list of URLs considered
and also add the oslo.config>=1.2.0a3 dependency:

  $> pip install \
       -f http://.../oslo.config-1.2.0a3.tar.gz#egg=oslo.config-1.2.0a3 \
       'oslo.config>=1.2.0a3' \
       python-keystoneclient
  $> pip freeze | grep oslo.config
  oslo.config-1.2.0a3
  $> python -c 'from oslo.config.cfg import DeprecatedOpt'

This is actually exactly the semantics we want and we go to great
lengths in pbr to get these semantics while using a single tarball URL.
The only downside to this --find-links strategy is that we gain an extra
line in our requirements.txt ... but it does work around the pip bug.

I think it makes sense for the requirements check to just ignore
--find-links lines for now like it does for URLs and -editable lines.
Using this method means we actually do require new versions of libraries
consumed this way to be approved into openstack/requirements first since
we have an explicit 'oslo.config>=1.2.0a3' listed rather than that being
derived from a URL.

It may make sense in future to have automation around checking which
find-links URLs are allowed ... but the same can be true for normal
dependency URLs. This change allows us to move forward and use latest
oslo.config in Nova, Neutron, etc. without falling foul of the pip bug.

Change-Id: I6f3eb5fd2c75615d9a1cae172aed859b36b27d4c

modules/jenkins/files/slave_scripts/project-requirements-change.py

@@ -41,11 +41,10 @@ def read_requirements(self, fn):
         for line in open(fn):
             line = line[:line.find('#')]
             line = line.strip()
-            if not line:
-                continue
-            if line.startswith('http'):
-                continue
-            if line.startswith('-e'):
+            if (not line or
+                line.startswith('http') or
+                line.startswith('-e') or
+                line.startswith('-f')):
                 continue
             req = pkg_resources.Requirement.parse(line)
             if req.project_name.lower() in self.reqs:

modules/openstack_project/files/jenkins_job_builder/config/requirements.yaml

@@ -15,7 +15,7 @@
             if [ -e $FILE ]
             then
               # Ignore lines beginning with https?:// just as the mirror script does.
-              sed -e '/https\?:\/\//d' $FILE > $FILE.clean
+              sed -e '/^https\?:\/\//d' $FILE > $FILE.clean
               PIP_ARGS="$PIP_ARGS -r $FILE.clean"
             fi
           done