Updated the py_pkgs lookup plugin for multi source

The py_pkgs lookup plugin will now handle multiple sources. This change
was needed to enable the system to allow for overrides of repo_package
resources and general python packaging as found throughout the stack.
The module has precedence as it loads / searches for packages and he item
loaded last is the one that has the most precedence.

The repo-build play has been updated to now search for compatible packages
in the main playbook directory, the root ansible role directory and the
user_variables file.

Documentation has been added regarding this capability and how to override
and set items to meet the needs of the deployment.

Closes-Bug: #1510575
Implements: blueprint independent-role-repositories
Change-Id: Ib7cda14db18c0ca58bb5ac495d1c201812cf0f48
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
(cherry picked from commit 2f35e36)

doc/source/developer-docs/extending.rst

@@ -139,6 +139,47 @@ This module has been `submitted for consideration`_ into Ansible Core.
 .. _Install Guide: ../install-guide/configure-openstack.html
 .. _submitted for consideration: https://github.com/ansible/ansible/pull/12555
 
+
+Build the environment with additional python packages
++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+The system will allow you to install and build any package that is a python
+installable. The repository infrastructure will look for and create any
+git based or PyPi installable package. When the package is built the repo-build
+role will create the sources as Python wheels to extend the base system and
+requirements.
+
+While the packages pre-built in the repository-infrastructure are comprehensive,
+it may be needed to change the source locations and versions of packages to suit
+different deployment needs. Adding additional repositories as overrides is as
+simple as listing entries within the variable file of your choice. Any
+``user_.*.yml`` file within the "/etc/openstack_deployment" directory will work
+to facilitate the addition of a new packages.
+
+
+.. code-block:: yaml
+
+    swift_git_repo: https://private-git.example.org/example-org/swift
+    swift_git_install_branch: master
+
+
+Additional lists of python packages can also be overridden using a ``user_.*.yml``
+variable file.
+
+.. code-block:: yaml
+
+    swift_requires_pip_packages:
+      - virtualenv
+      - virtualenv-tools
+      - python-keystoneclient
+      - NEW-SPECIAL-PACKAGE
+
+
+Once the variables are set call the play ``repo-build.yml`` to build all of the
+wheels within the repository infrastructure. When ready run the target plays to
+deploy your overridden source code.
+
+
 Module documentation
 ++++++++++++++++++++
 

playbooks/defaults/repo_packages/readme.rst

@@ -19,9 +19,15 @@ For the sake of anyone else editing this file:
   * If you add clients to this file please do so in alphabetical order.
   * Every entry should be name spaced with the name of the client followed by an "_"
 
+Repository data can be set in any of the following locations by default.
+    - <MAIN REPO LOCATION>
+    - /etc/ansible/roles
+    - /etc/openstack_deploy
+
 The basic structure of all of these files:
   * git_repo: ``string`` URI to the git repository to clone from.
   * git_fallback_repo: ``string`` URI to an alternative git repository to clone from when **git_repo** fails.
   * git_dest: ``string`` full path to place a cloned git repository. This will normally incorporate the **repo_path** variable for consistency purposes.
   * git_install_branch: ``string`` branch, tag or SHA of a git repository to clone into.
   * git_repo_plugins: ``list`` of ``hashes`` with keys: path, package | This is used to install additional packages which may be installable from the same base repository.
+  * git_package_name: ``string`` that will override the "egg" name given for the repo.

playbooks/plugins/filters/osa-filters.py

@@ -27,6 +27,28 @@
 """
 
 
+def _pip_requirement_split(requirement):
+    version_descriptors = "(>=|<=|>|<|==|~=|!=)"
+    requirement = requirement.split(';')
+    requirement_info = re.split(r'%s\s*' % version_descriptors, requirement[0])
+    name = requirement_info[0]
+    marker = None
+    if len(requirement) > 1:
+        marker = requirement[1]
+    versions = None
+    if len(requirement_info) > 1:
+        versions = requirement_info[1]
+
+    return name, versions, marker
+
+
+def _lower_set_lists(list_one, list_two):
+
+    _list_one = set([i.lower() for i in list_one])
+    _list_two = set([i.lower() for i in list_two])
+    return _list_one, _list_two
+
+
 def bit_length_power_of_2(value):
     """Return the smallest power of 2 greater than a numeric value.
 
@@ -120,17 +142,33 @@ def pip_requirement_names(requirements):
     :return: ``str``
     """
 
-    version_descriptors = "(>=|<=|>|<|==|~=|!=)"
     named_requirements = list()
     for requirement in requirements:
-        requirement = requirement.split(';')[0]
-        name = re.split(r'%s\s*' % version_descriptors, requirement)[0]
+        name = _pip_requirement_split(requirement)[0]
         if name and not name.startswith('#'):
             named_requirements.append(name.lower())
 
     return sorted(set(named_requirements))
 
 
+def pip_constraint_update(list_one, list_two):
+
+    _list_one, _list_two = _lower_set_lists(list_one, list_two)
+    _list_one, _list_two = list(_list_one), list(_list_two)
+    for item2 in _list_two:
+        item2_name, item2_versions, _ = _pip_requirement_split(item2)
+        if item2_versions:
+            for item1 in _list_one:
+                if item2_name == _pip_requirement_split(item1)[0]:
+                    item1_index = _list_one.index(item1)
+                    _list_one[item1_index] = item2
+                    break
+            else:
+                _list_one.append(item2)
+
+    return sorted(_list_one)
+
+
 def splitlines(string_with_lines):
     """Return a ``list`` from a string with lines."""
 
@@ -139,8 +177,7 @@ def splitlines(string_with_lines):
 
 def filtered_list(list_one, list_two):
 
-    _list_one = set([i.lower() for i in list_one])
-    _list_two = set([i.lower() for i in list_two])
+    _list_one, _list_two = _lower_set_lists(list_one, list_two)
     return list(_list_one-_list_two)
 
 
@@ -199,6 +236,7 @@ def filters():
             'netorigin': get_netorigin,
             'string_2_int': string_2_int,
             'pip_requirement_names': pip_requirement_names,
+            'pip_constraint_update': pip_constraint_update,
             'splitlines': splitlines,
             'filtered_list': filtered_list,
             'git_link_parse': git_link_parse,