feat(native_pip)!: switch from shelling out to pip to using native pi… (

#6) * feat(native_pip)!: switch from shelling out to pip to using native pip module * Adds "required_by" field * Adjusting tests to be more forgiving * Updates tests * Updates tests * Using argparse instead of env variable for path * Update README
Justintime50 · May 13, 2021 · 5fa1832 · 5fa1832
1 parent d9304cc
commit 5fa1832
Show file tree

Hide file tree

Showing 6 changed files with 161 additions and 169 deletions.
diff --git a/.coveragerc b/.coveragerc
@@ -2,3 +2,4 @@
 exclude_lines =
     if __name__ == '__main__':
     main()
+    PipTreeCli()
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,15 @@
 # CHANGELOG
 
+## v1.0.0 (2021-05-13)
+
+* Switched from shelling out to Pip to using the internal Pip API natively via Python (closes #4 and closes #2), this change makes the previous ~1 minute operation now take ~1 second!
+* Adds `updated` field indicating when the package was installed or updated (closes #5)
+* The `requires` and `required-by` keys are now lists instead of comma separated strings, they also include the version the requirements are pinned to
+* Using `argparse` instead of environment variable to specify path to site-packages
+* Separated out code better into classes and additional functions
+* 100% code coverage
+* Converted classmethods to staticmethods
+
 ## v0.5.0 (2020-11-24)
 
 * Removes pip version warnings from output

diff --git a/README.md b/README.md
@@ -13,7 +13,7 @@ Get the dependency tree of your Python virtual environment via Pip.
 
 </div>
 
-There is no simple, native way to get the dependency tree of a Python virtual environment using Pip as the package manager. Pip Tree fixes this problem by retrieving every package from your virtual environment and returning the packages it depends on as well as what depends on that package. These results will print to console.
+There is no simple, native way to get the dependency tree of a Python virtual environment using the Pip package manager for Python. Pip Tree fixes this problem by retrieving every package from your virtual environment and returning a list of JSON objects that include the package name, version installed, date updated, and which packages are required by each package (the tree).
 
 ## Install
 
@@ -30,57 +30,65 @@ make help
 
 ## Usage
 
-Invoke Pip Tree as a script and pass an optional pip path as an environment variable (great for per-project virtual environments). If no optional pip path is passed, then Pip Tree will attempt to use the system `pip3` installation.
-
-```bash
-PIP_PATH="path/to/my_project/venv/bin/pip" pip-tree
 ```
+Usage:
+    pip-tree --path "path/to/my_project/venv/lib/python3.9/site-packages"
 
-You can also import Pip Tree as a package and build custom logic for your needs. Pip Tree will return an array of json objects, each containing the name, version, packages required by the package, and what packages requires that package.
+Options:
+    -h, --help            show this help message and exit
+    -p PATH, --path PATH  The path to the site-packages directory of a Python virtual environment.
+```
 
-Set an optional pip path as an environment variable: `PIP_PATH="path/to/my_project/venv/bin/pip"`
+**Package**
 
 ```python
 from pip_tree import PipTree
 
-dependency_tree = PipTree.generate_dependency_tree()
+path = 'path/to/my_project/venv/lib/python3.9/site-packages'
 
-print(dependency_tree)
+package_list = PipTree.get_pip_package_list(path)
+for package in package_list:
+    package_object = PipTree.get_package_object(package)
+    package_details = PipTree.get_package_details(package_object)
+    print(package_details.project_name)
 ```
 
 **Sample Output**
 
 ```json
-Generating Pip Tree Report for "path/to/my_project/venv/bin/pip"...
+Generating Pip Tree Report...
 
 [
     {
-        "name": "aiohttp",
-        "version": "3.6.2",
-        "requires": "async-timeout, multidict, attrs, yarl, chardet",
-        "required-by": "slackclient"
-    },
-    {
-        "name": "astroid",
-        "version": "2.4.2",
-        "requires": "six, wrapt, lazy-object-proxy",
-        "required-by": ""
+        "name": "docopt",
+        "version": "0.6.2",
+        "updated": "2021-05-12",
+        "requires": []
     },
     {
-        "name": "async-timeout",
-        "version": "3.0.1",
-        "requires": "",
-        "required-by": "aiohttp"
+        "name": "flake8",
+        "version": "3.9.2",
+        "updated": "2021-05-12",
+        "requires": [
+            "mccabe<0.7.0,>=0.6.0",
+            "pyflakes<2.4.0,>=2.3.0",
+            "pycodestyle<2.8.0,>=2.7.0"
+        ]
     },
     {
-        "name": "attrs",
-        "version": "19.3.0",
-        "requires": "",
-        "required-by": "aiohttp"
+        "name": "Flask",
+        "version": "2.0.0",
+        "updated": "2021-05-12",
+        "requires": [
+            "itsdangerous>=2.0",
+            "click>=7.1.2",
+            "Werkzeug>=2.0",
+            "Jinja2>=3.0"
+        ]
     }
 ]
 
-Pip Tree report complete! 40 dependencies found for "path/to/my_project/venv/bin/pip".
+Pip Tree report complete! 40 dependencies found for "path/to/my_project/venv/lib/python3.9/site-packages".
 ```
 
 ## Development
@@ -98,5 +106,4 @@ make coverage
 
 ## Attribution
 
-- [GitHub Issue](https://github.com/pypa/pip/issues/5261#issuecomment-388173430) that helped with the refactor to Python
 - Icons made by <a href="https://www.flaticon.com/authors/freepik" title="Freepik">Freepik</a> from <a href="https://www.flaticon.com/" title="Flaticon"> www.flaticon.com</a>
diff --git a/pip_tree/tree.py b/pip_tree/tree.py
@@ -1,84 +1,115 @@
+import argparse
+import datetime
 import json
 import os
-import subprocess
-from email.parser import BytesHeaderParser
+import re
+import time
 
-PIP_PATH = os.getenv('PIP_PATH', 'pip3')
-TIMEOUT = 15
+import pkg_resources
 
 
-class PipTree():
-    @classmethod
-    def _generate_console_output(cls):
+class PipTreeCli():
+    def __init__(self):
+        parser = argparse.ArgumentParser(
+            description=(
+                'Get the dependency tree of your Python virtual environment via Pip.'
+            )
+        )
+        parser.add_argument(
+            '-p',
+            '--path',
+            required=True,
+            help='The path to the site-packages directory of a Python virtual environment.',
+        )
+        parser.parse_args(namespace=self)
+
+    def generate_console_output(self):
         """Take the output of the dependency tree and print to console.
         """
-        print(f'Generating Pip Tree report for "{PIP_PATH}"...')
-        console_output, number_of_dependencies = cls.generate_dependency_tree()
-        print(json.dumps(console_output, indent=4))
-        print(f'Pip Tree report complete! {number_of_dependencies} dependencies found for "{PIP_PATH}".')
-
-    @classmethod
-    def generate_dependency_tree(cls):
-        """Generate the dependency tree of your pip virtual environment
-        and print to console.
+        print('Generating Pip Tree report...')
+        final_output, package_count = PipTree.generate_pip_tree(self.path)
+        print(json.dumps(final_output, indent=4))
+        print(f'Pip Tree report complete! {package_count} dependencies found for "{self.path}".')
+
+
+class PipTree():
+    @staticmethod
+    def generate_pip_tree(path):
+        """Generate the Pip Tree of the virtual environment specified.
         """
-        package_list = cls.get_pip_package_list()
-        dependency_tree, number_of_dependencies = cls.get_package_dependencies(package_list)
-        return dependency_tree, number_of_dependencies
+        pip_tree_results = []
+        required_by_dict = {}
+        package_count = 0
+        packages = PipTree.get_pip_package_list(path)
+
+        for package in packages:
+            package_object = PipTree.get_package_object(package)
+            package_details = PipTree.get_package_details(package_object)
+            PipTree.generate_reverse_requires_field(required_by_dict, package_details)
+            pip_tree_results.append(package_details)
+            package_count += 1
+
+        # Append the `required_by` field to each record
+        for item in pip_tree_results:
+            item['required_by'] = sorted(required_by_dict.get(item['name'], []))
+
+        final_output = sorted(pip_tree_results, key=lambda k: k['name'].lower())
 
-    @classmethod
-    def get_pip_package_list(cls):
+        return final_output, package_count
+
+    @staticmethod
+    def get_pip_package_list(path):
         """Get the pip package list of the virtual environment.
+
+        Must be a path like: /project/venv/lib/python3.9/site-packages
         """
-        try:
-            command = f'{PIP_PATH} list --format=json --disable-pip-version-check --isolated --no-input'
-            package_list_output = subprocess.check_output(
-                command,
-                stdin=None,
-                stderr=None,
-                shell=True,
-                timeout=TIMEOUT
-            )
-        except subprocess.TimeoutExpired:
-            raise subprocess.TimeoutExpired(command, TIMEOUT)
-        except subprocess.CalledProcessError:
-            raise subprocess.CalledProcessError(127, command)
-        return json.loads(package_list_output)
-
-    @classmethod
-    def get_package_dependencies(cls, package_list):
-        """Get a single package dependencies and return a json object
+        packages = pkg_resources.find_distributions(path)
+        return packages
+
+    @staticmethod
+    def get_package_object(package):
+        """Returns a package object from Pip.
         """
-        final_list = []
-        package_count = 0
-        for package in package_list:
-            try:
-                command = f'{PIP_PATH} show {package["name"]} --disable-pip-version-check  --isolated --no-input'
-                package_output = subprocess.check_output(
-                    command,
-                    stdin=None,
-                    stderr=None,
-                    shell=True,
-                    timeout=TIMEOUT
+        package_object = pkg_resources.get_distribution(package)
+        return package_object
+
+    @staticmethod
+    def get_package_details(package):
+        """Build a dictionary of details for a package from Pip.
+        """
+        package_update_at = time.ctime(os.path.getctime(package.location))
+        requires_list = [sorted(str(requirement) for requirement in package.requires())]
+        package_details = {
+            'name': package.project_name,
+            'version': package.version,
+            'updated': datetime.datetime.strptime(package_update_at, "%a %b %d %H:%M:%S %Y").strftime("%Y-%m-%d"),
+            'requires': [item for sublist in requires_list for item in sublist],
+        }
+        return package_details
+
+    @staticmethod
+    def generate_reverse_requires_field(required_by_dict, package_details):
+        """Generate a reversed list from the `requires` fields and create a collection
+        of each `required_by` fields so each package can show what it's required_by
+        """
+        requires_list = [item for item in package_details['requires']]
+        for required_by_package in requires_list:
+            word = re.compile(r'^(\w)+')
+            required_by_package_name = word.match(required_by_package).group()
+
+            if required_by_dict.get(required_by_package_name):
+                required_by_dict[required_by_package_name].append(package_details['name'])
+            else:
+                required_by_dict.update(
+                    {
+                        required_by_package_name: [package_details['name']]
+                    }
                 )
-            except subprocess.TimeoutExpired:
-                raise subprocess.TimeoutExpired(command, TIMEOUT)
-            except subprocess.CalledProcessError:
-                raise subprocess.CalledProcessError(127, command)
-            parsed_package_output = BytesHeaderParser().parsebytes(package_output)
-            final_package_output = {
-                'name': parsed_package_output['Name'],
-                'version': parsed_package_output['Version'],
-                'requires': parsed_package_output['Requires'],
-                'required-by': parsed_package_output['Required-by'],
-            }
-            final_list.append(final_package_output)
-            package_count += 1
-        return final_list, package_count
+        return required_by_dict
 
 
 def main():
-    PipTree._generate_console_output()
+    PipTreeCli().generate_console_output()
 
 
 if __name__ == '__main__':

diff --git a/setup.py b/setup.py
@@ -5,7 +5,7 @@
 
 setuptools.setup(
     name='pip-tree',
-    version='0.5.0',
+    version='1.0.0',
     description='Get the dependency tree of your Python virtual environment via Pip.',
     long_description=long_description,
     long_description_content_type="text/markdown",