Add type annotations #1761

chadrik · 2024-05-19T17:38:29Z

This is a first pass at adding type annotations throughout the code-base. Mypy is not fully passing yet, but it's getting close.

Addresses #1631

linux-foundation-easycla · 2024-05-19T17:38:33Z

❌ - login: @chadrik / name: Chad Dombrova . The commit (c1d368a, fa8d564, 266c3a4, e48b4e2, 6863989, b9cc27c, 1d2418c, 19dc1db, dcc3e63, 614c1f6, ea70e38, 43aedad, 4fd0bac) is not authorized under a signed CLA. Please click here to be authorized. For further assistance with EasyCLA, please submit a support request ticket.

chadrik · 2024-05-19T17:43:55Z

src/rez/package_filter.py

@@ -327,7 +327,7 @@ class Rule(object):
    """Base package filter rule"""

    #: Rule name
-    name = None
+    name: str


This pattern appears on abstract base classes in a number of places, and we have 3 choices:

Option 1:

name: str | None = None

typically produces many spurious mypy errors, because the code assumes the value is a string everywhere, and the None-case is not handled.

Option 2:

name: str

only safe if we're sure that the name attribute is always accessed on concrete sub-classes, otherwise will result in an AttributeError since this does not actually create an attribute.

Option 3:

@property @abstractmethod def name(self) -> str: raise NotImplementedError

Provides us some guarantee that sub-classes actually implement name, but it cannot be accessed from the class, only from an instance.

chadrik · 2024-05-19T17:45:40Z

src/rez/package_order.py

-    def __eq__(self, other):
+    def __eq__(self, other: object) -> bool:
+        if not isinstance(other, FallbackComparable):
+            return NotImplemented


mypy strongly prefers that comparison functions define the type of other as object. by returning NotImplemented we allow other other object to handle equality if it implements it. This should not change the behavior as long as other is always FallbackComparable within Rez.

chadrik · 2024-05-19T17:47:39Z

src/rez/package_resources.py

-    pass
+
+    def iter_packages(self) -> Iterator[Package]:
+        raise NotImplementedError


All subclasses of PackageFamilyResource implement this method, so it appears to be considered an abstractmethod of PackageFamilyResource. Defining it simplifies some type annotation problems.

chadrik · 2024-05-19T17:48:10Z

src/rez/package_resources.py

+    @property
+    @abstractmethod
+    def parent(self) -> PackageRepositoryResource:
+        raise NotImplementedError


same situation here as with PackageFamilyResource.iter_packages.

chadrik · 2024-05-19T17:48:19Z

src/rez/package_resources.py

+    @property
+    @abstractmethod
+    def parent(self) -> PackageRepositoryResource:
+        raise NotImplementedError


here as well.

chadrik · 2024-05-19T17:49:24Z

src/rez/packages.py

@@ -319,6 +326,7 @@ def get_variant(self, index=None):
        for variant in self.iter_variants():
            if variant.index == index:
                return variant
+        return None


mypy prefer explicit return None statements

chadrik · 2024-05-19T17:50:06Z

src/rez/resolved_context.py

-                raise ResolvedContextError(
-                    "Cannot perform operation in a failed context")
-        return _check
-


mypy does not like these decorators defined at the class-level.

chadrik · 2024-05-19T18:08:22Z

Protocol and TypedDict are not available in the typing module until python 3.8.

We have a few options:

vendor typing_extensions
remove use of these typing classes until Rez drops support for python 3.7
I can create a mock of Protocol and trick mypy into using it which is safe because it has no runtime behavior. Doing the same thing for TypedDict is more complicated, but possible.

codecov · 2024-05-19T18:33:50Z

Codecov Report

Attention: Patch coverage is 84.77322% with 141 lines in your changes are missing coverage. Please review.

Project coverage is 58.27%. Comparing base (a13f7bb) to head (dcc3e63).

Files	Patch %	Lines
src/rez/package_order.py	72.72%	14 Missing and 4 partials ⚠️
src/rez/solver.py	93.63%	11 Missing and 3 partials ⚠️
src/rez/resolved_context.py	72.34%	11 Missing and 2 partials ⚠️
src/rez/version/_version.py	91.24%	8 Missing and 4 partials ⚠️
src/rez/package_resources.py	77.55%	9 Missing and 2 partials ⚠️
src/rez/packages.py	85.45%	7 Missing and 1 partial ⚠️
src/rez/build_process.py	75.00%	5 Missing and 2 partials ⚠️
src/rez/utils/data_utils.py	79.31%	4 Missing and 2 partials ⚠️
src/rez/utils/resources.py	76.92%	4 Missing and 2 partials ⚠️
src/rez/build_system.py	81.48%	4 Missing and 1 partial ⚠️
... and 18 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1761      +/-   ##
==========================================
- Coverage   58.41%   58.27%   -0.14%     
==========================================
  Files         126      126              
  Lines       17163    17278     +115     
  Branches     3506     3550      +44     
==========================================
+ Hits        10025    10069      +44     
- Misses       6473     6519      +46     
- Partials      665      690      +25

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

chadrik · 2024-05-20T05:34:07Z

I got bored and added lots more, particularly focused on the solver module. Once the solver module is complete, we can experiment with compiling it to a c-extension using mypyc, which could provide a big speed boost!

chadrik · 2024-05-21T03:28:27Z

I now have rez.solver compiling as a C-extension with all tests passing. I'm very interested to see how the performance compares. Does anyone want to volunteer to help put together a performance comparison? Are there any known complex collection of packages to test against?

chadrik · 2024-05-21T03:37:46Z

src/rez/package_order.py

-        self.dirty = True
-        return super().append(*args, **kwargs)
+    if not TYPE_CHECKING:
+        def append(self, *args, **kwargs):


Since this class inherits from list it's easier to rely on the type hints coming from that base class than to redefine them here, so we hide them by placing them behind not TYPE_CHECKING. In reality, the runtime value of TYPE_CHECKING is always False.

chadrik · 2024-05-21T03:40:32Z

src/rez/plugin_managers.py

+    def get_plugin_class(self, plugin_type: str, plugin_name: str, expected_type: type[T]) -> type[T]:
+        pass
+
+    def get_plugin_class(self, plugin_type, plugin_name, expected_type=None):
        """Return the class registered under the given plugin name."""
        plugin = self._get_plugin_type(plugin_type)


Added a new argument here to validate the returned result. This provides both runtime and static validation.

chadrik · 2024-05-21T03:41:43Z

src/rez/resolved_context.py

        """Perform a package resolve, and store the result.

        Args:
-            package_requests (list[typing.Union[str, PackageRequest]]): request
+            package_requests (list[typing.Union[str, Requirement]]): request


I noticed that everywhere that we've documented types as PackageRequest, they appear to actually be Requirement. I'm not sure if there any real-world exceptions to this.

chadrik · 2024-05-21T03:42:10Z

src/rez/resolved_context.py

@@ -884,7 +912,7 @@ def _rt(t):
            return

        _pr("resolved packages:", heading)
-        rows = []
+        rows3: list[tuple[str, str, str]] = []


you can't redefine types with mypy, so you need to use new variable names.

chadrik · 2024-05-21T03:42:54Z

src/rez/system.py

@@ -88,7 +88,7 @@ def shell(self):
                    args = ['ps', '-o', 'args=', '-p', str(parent_pid)]
                    proc = sp.Popen(args, stdout=sp.PIPE)
                    output = proc.communicate()[0]
-                    shell = os.path.basename(output.strip().split()[0]).replace('-', '')
+                    shell = os.path.basename(output.decode().strip().split()[0]).replace('-', '')


output is bytes in python3, so need to call decode()

chadrik · 2024-05-21T03:43:31Z

src/rez/utils/data_utils.py

+if TYPE_CHECKING:
+    cached_property = property
+else:
+    class cached_property(object):


it's much easier to pretend that cached_property is property than to type hint all the subtleties of a descriptor.

chadrik · 2024-05-21T03:45:46Z

src/rez/solver.py

+        self.depth_counts: dict
+        self.solve_begun: bool
+        self.solve_time: float
+        self.load_time: float


these values are never actually None, so we define the types here, and their values are assigned in _init()

chadrik · 2024-05-21T03:48:40Z

src/rez/solver.py

@@ -74,9 +97,6 @@ class SolverStatus(Enum):
    cyclic = ("The solve contains a cycle.", )
    unsolved = ("The solve has started, but is not yet solved.", )

-    def __init__(self, description):
-        self.description = description
-


mypyc did not like adding a new __init__ for Enum, so it was simple enough to replace this attribute with a call to the enum value.

chadrik · 2024-05-21T03:49:58Z

src/rez/solver.py

        """Reset the solver, removing any current solve."""
        if not self.request_list.conflict:
-            phase = _ResolvePhase(self.request_list.requirements, solver=self)
+            phase = _ResolvePhase(solver=self)


This appears to be a bug: _ResolvePhase only takes one argument. mypy to the rescue.

chadrik · 2024-05-21T04:48:54Z

I found rez-benchmark. Interestingly, rez is slower with the compiled rez.solver. It could be because there are many modules and classes used by rez.solver which have not been compiled.

I probably won't have time to dig into this much more, but once this PR is merged I'll make a new PR with the changes necessary for people to test the compiled version of rez.

chadrik · 2024-05-21T18:27:53Z

Note: this PR likely invalidates #1745

chadrik · 2024-06-05T18:04:33Z

@instinct-vfx Can you or someone from the Rez group have a look at this, please?

Add type annotations

fa8d564

chadrik requested a review from a team as a code owner May 19, 2024 17:38

chadrik commented May 19, 2024

View reviewed changes

chadrik added 2 commits May 19, 2024 11:09

Fixes

266c3a4

fixes

c1d368a

Lots more typing

1d2418c

chadrik added 3 commits May 20, 2024 08:59

fixes

6863989

solver is now nearly fully complete

e48b4e2

fixes

b9cc27c

chadrik commented May 21, 2024

View reviewed changes

chadrik added 2 commits May 20, 2024 20:57

lint fixes

ea70e38

lint fixes

614c1f6

chadrik added 4 commits May 21, 2024 22:48

More types

4fd0bac

fixes

19dc1db

More types

43aedad

fixes

dcc3e63

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add type annotations #1761

Add type annotations #1761

chadrik commented May 19, 2024

linux-foundation-easycla bot commented May 19, 2024 •

edited

chadrik May 19, 2024 •

edited

chadrik May 19, 2024

chadrik May 19, 2024

chadrik May 19, 2024

chadrik May 19, 2024

chadrik May 19, 2024

chadrik May 19, 2024

chadrik commented May 19, 2024

codecov bot commented May 19, 2024 •

edited

chadrik commented May 20, 2024

chadrik commented May 21, 2024

chadrik May 21, 2024

chadrik May 21, 2024

chadrik May 21, 2024

chadrik May 21, 2024

chadrik May 21, 2024

chadrik May 21, 2024

chadrik May 21, 2024

chadrik May 21, 2024

chadrik May 21, 2024

chadrik commented May 21, 2024

chadrik commented May 21, 2024

chadrik commented Jun 5, 2024

Add type annotations #1761

Are you sure you want to change the base?

Add type annotations #1761

Conversation

chadrik commented May 19, 2024

linux-foundation-easycla bot commented May 19, 2024 • edited

chadrik May 19, 2024 • edited

Choose a reason for hiding this comment

Option 1:

Option 2:

Option 3:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chadrik commented May 19, 2024

codecov bot commented May 19, 2024 • edited

Codecov Report

chadrik commented May 20, 2024

chadrik commented May 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chadrik commented May 21, 2024

chadrik commented May 21, 2024

chadrik commented Jun 5, 2024

linux-foundation-easycla bot commented May 19, 2024 •

edited

chadrik May 19, 2024 •

edited

codecov bot commented May 19, 2024 •

edited