JVM lockfile invalidation headers and verification #14185

chrisjrn · 2022-01-18T17:35:26Z

This adds support for JVM lockfile invalidation headers. Most of the infrastructure is shared with the equivalent Python support code.

Closes #13373

Notes for reviewers:

The lockfile format now adds a header of the same form as our Python lockfiles (which have been thoroughly bikeshedded at this stage). Once you've seen one lockfile with contents in the valid_for_requirements field, you've seen them all, and feel free to skip.

There's no new tests, but both the tool and user lockfile invalidation code seems to be hit pretty thoroughly by all the existing tests (note the numerous failures in the commit history :)). If there's any specific behaviour in lockfile_metadata.py that you feel needs testing, please note it in the review.

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

…; deserializes more centrally # Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

…eading toml lockfiles # Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

…e to avoid a circuilar import later # Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

…e to avoid a circuilar import later [ci skip-rust] [ci skip-build-wheels] # Conflicts: # src/python/pants/jvm/goals/coursier.py # src/python/pants/jvm/resolve/coursier_fetch.py

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

…ckfile-invalidation [ci skip-rust]

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

# Conflicts: # 3rdparty/jvm/global_scalac_plugins.lock # 3rdparty/jvm/testprojects.lockfile # src/python/pants/backend/codegen/avro/java/avro-tools.default.lockfile.txt # src/python/pants/backend/codegen/avro/java/rules.py # src/python/pants/backend/codegen/protobuf/scala/rules.py # src/python/pants/backend/codegen/protobuf/scala/scalapbc.default.lockfile.txt # src/python/pants/backend/codegen/thrift/scrooge/rules.py # src/python/pants/backend/codegen/thrift/scrooge/scrooge.default.lockfile.txt # src/python/pants/backend/java/dependency_inference/java_parser.lockfile # src/python/pants/backend/java/lint/google_java_format/google_java_format.default.lockfile.txt # src/python/pants/backend/java/lint/google_java_format/rules.py # src/python/pants/backend/scala/dependency_inference/scala_parser.lockfile # src/python/pants/backend/scala/lint/scalafmt/rules.py # src/python/pants/backend/scala/lint/scalafmt/scalafmt.default.lockfile.txt # src/python/pants/backend/scala/subsystems/scalatest.default.lockfile.txt # src/python/pants/backend/scala/test/scalatest.py # src/python/pants/jvm/goals/coursier.py # src/python/pants/jvm/resolve/coursier_fetch.py # src/python/pants/jvm/resolve/jvm_tool.py # src/python/pants/jvm/test/junit.default.lockfile.txt # src/python/pants/jvm/test/junit.py # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

chrisjrn · 2022-01-18T17:42:23Z

src/python/pants/jvm/goals/coursier.py

+    resolved_serialized_lockfile = resolved_lockfile.to_serialized()
+    metadata = JVMLockfileMetadata.new(requirements)
+    resolved_serialized_lockfile = metadata.add_header_to_lockfile(
+        resolved_serialized_lockfile, regenerate_command="./pants generate-lockfiles"


TODO: check if this is correct after Eric's changes

This will be correct once the code gets moved, but presently it isn't.

I'm close to putting up this change, so this is fine.

chrisjrn · 2022-01-18T17:44:19Z

src/python/pants/jvm/resolve/coursier_fetch.py


        contents = toml.loads(lockfile_str)
        entries = tuple(
            CoursierLockfileEntry.from_json_dict(entry) for entry in (contents["entries"])
        )
+        metadata = JVMLockfileMetadata.from_lockfile(lockfile_bytes)


This was written before I factored out the TOML-parsing code in #14175, this will fail if the metadata header block is not in place.

Unsure if we want to add a super-quick deprecation cycle here, but I don't believe we've put the contents of #14175 in a release yet?

We haven't, no need to handle deprecation.

chrisjrn · 2022-01-18T17:44:51Z

src/python/pants/jvm/resolve/jvm_tool.py

+
+    if lockfile.metadata and not lockfile.metadata.is_valid_for(requirements):
+        raise ValueError(
+            f"Lockfile for {request.options_scope} was generated with different "


Error handling code needs to be better (at least as good as Python's :))

For context, the Python tool lockfile error message code is a hundred lines of code with carefully specified error messages. We should probably tackle this as a separate issue, as this PR is already pretty big.

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

3rdparty/jvm/global_scalac_plugins.lock

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

chrisjrn · 2022-01-19T16:11:08Z

src/python/pants/jvm/goals/coursier_integration_test.py

-    assert (
-        Path(rule_runner.build_root, "3rdparty/jvm/default.lock").read_bytes()
-        == HAMCREST_EXPECTED_LOCKFILE.to_serialized()
+    compare_lockfiles(


This changes the test -- we're now deserializing the file on disk and comparing to the structure we defined in the test, rather than checking that the structure is byte-for-byte correct.

Much better :)

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

chrisjrn · 2022-01-19T16:36:02Z

src/python/pants/jvm/goals/coursier.py

    lockfile_path = jvm.resolves[request.resolve]

    # If the lockfile hasn't changed, don't overwrite it.
    existing_lockfile_digest_contents = await Get(DigestContents, PathGlobs([lockfile_path]))
    if (
        existing_lockfile_digest_contents
-        and resolved_lockfile_serialized == existing_lockfile_digest_contents[0].content
+        and resolved_serialized_lockfile == existing_lockfile_digest_contents[0].content


Looks like I chose different variable names in #14175. Whoops :)

chrisjrn · 2022-01-19T16:38:33Z

src/python/pants/jvm/resolve/coursier_fetch.py

@@ -449,13 +456,22 @@ async def fetch_with_coursier(request: CoursierFetchRequest) -> FallibleClasspat
    # TODO: Loading this per JvmArtifact.
    lockfile = await Get(CoursierResolvedLockfile, CoursierResolveKey, request.resolve)



NOTE: The following code is for validating user lockfiles, and is necessarily done per JvmArtifact, because of how we bubble a resolved classpath up to the consuming code.

That'd be good to put in a comment. To clarify, that is because this is the rule for CoursierFetchRequest?

chrisjrn · 2022-01-19T16:39:08Z

src/python/pants/jvm/resolve/coursier_fetch.py

+    requirement = ArtifactRequirement.from_jvm_artifact_target(request.component.representative)
+
+    if lockfile.metadata and not lockfile.metadata.is_valid_for([requirement]):
+        raise ValueError(


ValueError seems to be the exception we've been using in this part of the codebase. Let me know if there's anything more appropriate here.

We could probably define a new error type, but NBD.

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

chrisjrn · 2022-01-19T16:46:22Z

Tagging @Eric-Arellano for familiarity with how we handle lockfiles on the Python side, and @stuhood for signoff on actual JVM functionality

Eric-Arellano · 2022-01-19T20:12:43Z

src/python/pants/backend/java/lint/google_java_format/rules.py

 from pants.util.logging import LogLevel
 from pants.util.strutil import pluralize

+logger = logging.getLogger(__name__)


Nit: not necessary

Eric-Arellano · 2022-01-19T20:13:51Z

src/python/pants/jvm/goals/coursier.py

+    resolved_serialized_lockfile = resolved_lockfile.to_serialized()
+    metadata = JVMLockfileMetadata.new(requirements)
+    resolved_serialized_lockfile = metadata.add_header_to_lockfile(
+        resolved_serialized_lockfile, regenerate_command="./pants generate-lockfiles"


I'm close to putting up this change, so this is fine.

Eric-Arellano · 2022-01-19T20:14:50Z

src/python/pants/jvm/goals/coursier_integration_test.py

-    assert (
-        Path(rule_runner.build_root, "3rdparty/jvm/default.lock").read_bytes()
-        == HAMCREST_EXPECTED_LOCKFILE.to_serialized()
+    compare_lockfiles(


Much better :)

Eric-Arellano · 2022-01-19T20:15:27Z

src/python/pants/jvm/resolve/coursier_fetch.py


        contents = toml.loads(lockfile_str)
        entries = tuple(
            CoursierLockfileEntry.from_json_dict(entry) for entry in (contents["entries"])
        )
+        metadata = JVMLockfileMetadata.from_lockfile(lockfile_bytes)


We haven't, no need to handle deprecation.

Eric-Arellano · 2022-01-19T20:16:12Z

src/python/pants/jvm/resolve/coursier_fetch.py

@@ -449,13 +456,22 @@ async def fetch_with_coursier(request: CoursierFetchRequest) -> FallibleClasspat
    # TODO: Loading this per JvmArtifact.
    lockfile = await Get(CoursierResolvedLockfile, CoursierResolveKey, request.resolve)



That'd be good to put in a comment. To clarify, that is because this is the rule for CoursierFetchRequest?

Eric-Arellano · 2022-01-19T20:16:30Z

src/python/pants/jvm/resolve/coursier_fetch.py

+    requirement = ArtifactRequirement.from_jvm_artifact_target(request.component.representative)
+
+    if lockfile.metadata and not lockfile.metadata.is_valid_for([requirement]):
+        raise ValueError(


We could probably define a new error type, but NBD.

src/python/pants/jvm/resolve/jvm_tool.py

Eric-Arellano · 2022-01-19T20:24:01Z

src/python/pants/backend/java/lint/google_java_format/rules.py

-                lockfiles=(tool.resolved_lockfile(),),
+                lockfiles=(lockfile,),


Oh huh, this was violating the engine by using open() inside lockfile_content

Eric-Arellano · 2022-01-19T20:30:46Z

src/python/pants/jvm/resolve/jvm_tool.py

+    lockfile = request.lockfile
+    requirements = await Get(
+        ArtifactRequirements,
+        GatherJvmCoordinatesRequest(
+            request.artifact_inputs, f"[{request.options_scope}].artifacts"
+        ),
+    )


I'm trying to reason about if this should be requesting JvmLockfileRequest or not, and then looking at the request.artifacts. It seems wrong for us to be computing the two requirements in distinct ways. But I'm having a hard time following everything

Yeah, I had some difficulty here. Let's discuss what a refactor would look like elsewhere.

Yea, not a blocker, but this is odd. I think that I might have caused some of the oddity though, because global scalac_plugins have a lockfile, but don't have an associated subsystem. And so ValidatedJvmToolLockfileRequest cannot take a subsystem as an argument, because not all requests have one.

Eric-Arellano · 2022-01-19T20:36:37Z

Would it make sense to have this PR only add the metadata to the lockfiles, but not yet do anything with them?

I'm having a hard time reasoning about how the validation happens, and it might change once I land very soon unifying coursier-resolve with generate-lockfiles. Also, our consumption of lockfiles is currently broken because we are using open() rather than the engine.

I agree with your intuition that we would likely benefit from refactoring the JVM lockfile code. So I'm thinking it might make sense to do that before changing any consumption code.

Regardless of what we do there, the writing of metadata looks great and seems ready to land. Wdyt?

chrisjrn · 2022-01-19T21:10:22Z

I'm OK with adding the invalidation headers in a separate PR, but I'm hesitant to defer the remainder of the PR for long.

…ckfile-invalidation # Conflicts: # src/python/pants/jvm/goals/lockfile.py

Eric-Arellano · 2022-01-19T22:21:23Z

src/python/pants/jvm/resolve/lockfile_metadata.py

+    def _header_dict(self) -> dict[Any, Any]:
+        """Produce a dictionary to be serialized into the lockfile header.
+
+        Subclasses should call `super` and update the resulting dictionary.
+        """
+
+        d = super()._header_dict()
+        return d


Why override this?

You have to override it in subclasses, and you need to keep super in place. This override here is defensive work to make sure that people who eventually add things into _header_dict in this class don't do it the wrong way.

In general, I find that it is preferable in OO to avoid APIs that require overriding concrete methods, and to stick to completely abstract (or empty) methods or fields. But not a blocker.

chrisjrn · 2022-01-19T22:23:57Z

Following discussion with @Eric-Arellano, we're probably good to merge this as-is. Can you review please, @stuhood?

Eric-Arellano

As discussed over DM, there are some known issues:

Don't use tool.resolved_lockfiles()
Figure out how to get this working with scalac_plugins.py
Figure out how the consumption rule should be computing the ArtifactRequirements

But this gets us closer to where we want, and it's fine to fast-follow.

stuhood

Thanks!

It would be good to update the title to indicate that this is only for tool lockfiles (afaict).

stuhood · 2022-01-19T22:28:50Z

src/python/pants/core/util_rules/lockfile_metadata.py

 class LockfileScope(Enum):
+    JVM = "jvm"
    PYTHON = "python"


@Eric-Arellano : Not a blocker for this change, but this seems like something that should be computed from one of the @unions involved here?

Maybe which metadata class to create when generating a lockfile, but nothing else.

stuhood · 2022-01-19T22:36:19Z

src/python/pants/jvm/resolve/jvm_tool.py

+    lockfile = request.lockfile
+    requirements = await Get(
+        ArtifactRequirements,
+        GatherJvmCoordinatesRequest(
+            request.artifact_inputs, f"[{request.options_scope}].artifacts"
+        ),
+    )


Yea, not a blocker, but this is odd. I think that I might have caused some of the oddity though, because global scalac_plugins have a lockfile, but don't have an associated subsystem. And so ValidatedJvmToolLockfileRequest cannot take a subsystem as an argument, because not all requests have one.

stuhood · 2022-01-19T22:39:00Z

src/python/pants/jvm/resolve/lockfile_metadata.py

+    def _header_dict(self) -> dict[Any, Any]:
+        """Produce a dictionary to be serialized into the lockfile header.
+
+        Subclasses should call `super` and update the resulting dictionary.
+        """
+
+        d = super()._header_dict()
+        return d


In general, I find that it is preferable in OO to avoid APIs that require overriding concrete methods, and to stick to completely abstract (or empty) methods or fields. But not a blocker.

chrisjrn · 2022-01-19T22:43:20Z

@stuhood This has support for user lockfiles now, too.

stuhood · 2022-01-19T23:15:58Z

@stuhood This has support for user lockfiles now, too.

So it does! I think that that means that CoursierResolvedLockfile._coordinate_not_found should be unreachable now.

…thRequest` (#14241) To consume a tool lockfile, we before would: 1. Read the lockfile, either by: 1. Using `ValidatedJvmToolLockfileRequest`, which only works if it's a `JvmToolBase` 2. Using `GenerateJvmLockfile -> CoursierResolvedLockfile` rule, which we used with `scalac_plugins.py` 2. Pass the lockfile to `ToolClasspathRequest -> ToolClasspath` The rule in 1.2 was not using the validation added in #14185, and it was generally confusing to have two code paths. This unifies everything by having `ToolClasspathRequest -> ToolClasspath` now handle reading, validating, and then fetching the lockfile. Reading and validating a lockfile requires the same metadata we need for generating a lockfile with `generate-lockfiles` via `GenerateJvmLockfile`. So, we reuse that type for the sake of DRY. [ci skip-rust]

Christopher Neugebauer added 14 commits January 13, 2022 14:09

Adds JVMLockfileMetadata and related things

e5142c3

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Moves CoursierResolvedLockfile.from_json_dict to .from_serialized…

a274adf

…; deserializes more centrally # Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Serializes coursier lockfiles to toml, and adds initial support for r…

60b3f43

…eading toml lockfiles # Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Generated lockfiles now include invalidation headers

8595016

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

WIP

1536ac9

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Pulls Coordinate and ArtifactRequirements out into a separate fil…

df0c578

…e to avoid a circuilar import later # Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Pulls Coordinate and ArtifactRequirements out into a separate fil…

3ceac13

…e to avoid a circuilar import later [ci skip-rust] [ci skip-build-wheels] # Conflicts: # src/python/pants/jvm/goals/coursier.py # src/python/pants/jvm/resolve/coursier_fetch.py

Validate lockfiles in jvm_tool.py

31cc4b7

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Regenerates tool lockfiles into new toml format

53ffa56

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Validates lockfile headers for google java format

89b6f18

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

All JVM tools validate their lockfile on setup

37c464c

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Merge remote-tracking branch 'origin/main' into chrisjrn/13373-jvm-lo…

4129ee0

…ckfile-invalidation [ci skip-rust]

Updates JVM user lockfiles

0e83cf1

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

chrisjrn marked this pull request as draft January 18, 2022 17:35

chrisjrn commented Jan 18, 2022

View reviewed changes

Christopher Neugebauer added 5 commits January 18, 2022 10:35

Fix the tool lockfile generating code to actually execute

49e23b3

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

fmt

3c50573

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Fix tests

3434461

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Add validation of user lockfiles

410c115

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Fix tests

2a43202

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

chrisjrn changed the title ~~[WIP/DRAFT] JVM lockfile invalidation headers and verification (for tools only)~~ [WIP/DRAFT] JVM lockfile invalidation headers and verification Jan 18, 2022

Fixmore tests

9a6d8fb

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Eric-Arellano reviewed Jan 18, 2022

View reviewed changes

3rdparty/jvm/global_scalac_plugins.lock Show resolved Hide resolved

Christopher Neugebauer added 2 commits January 18, 2022 15:39

fmt

622332f

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Renames TCoursierResolvedLockfile to CoursierLockfileWrapper

65af690

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

chrisjrn commented Jan 19, 2022

View reviewed changes

Update error message to be less terse/useless

647c9f5

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Christopher Neugebauer added 2 commits January 19, 2022 08:29

Updates is_valid_for docstring

e987f2e

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

simplifies junit_test slightly

fc53b02

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

chrisjrn commented Jan 19, 2022

View reviewed changes

fix parenthesisation

b55f229

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

chrisjrn marked this pull request as ready for review January 19, 2022 16:45

chrisjrn requested review from stuhood and Eric-Arellano January 19, 2022 16:45

chrisjrn changed the title ~~[WIP/DRAFT] JVM lockfile invalidation headers and verification~~ JVM lockfile invalidation headers and verification Jan 19, 2022

chrisjrn mentioned this pull request Jan 19, 2022

[internal] Make JvmLockfileRequest generic #14201

Merged

Eric-Arellano reviewed Jan 19, 2022

View reviewed changes

Christopher Neugebauer added 2 commits January 19, 2022 13:15

Merge remote-tracking branch 'origin/main' into chrisjrn/13373-jvm-lo…

0dc2130

…ckfile-invalidation # Conflicts: # src/python/pants/jvm/goals/lockfile.py

Address code review floop

858f83e

Eric-Arellano reviewed Jan 19, 2022

View reviewed changes

Eric-Arellano approved these changes Jan 19, 2022

View reviewed changes

stuhood approved these changes Jan 19, 2022

View reviewed changes

chrisjrn merged commit e771911 into pantsbuild:main Jan 19, 2022

Eric-Arellano mentioned this pull request Jan 24, 2022

[internal] Inline ValidatedJvmToolLockfileRequest into ToolClasspathRequest #14241

Merged

		@@ -449,13 +456,22 @@ async def fetch_with_coursier(request: CoursierFetchRequest) -> FallibleClasspat
		# TODO: Loading this per JvmArtifact.
		lockfile = await Get(CoursierResolvedLockfile, CoursierResolveKey, request.resolve)

		lockfiles=(tool.resolved_lockfile(),),
		lockfiles=(lockfile,),

JVM lockfile invalidation headers and verification #14185

JVM lockfile invalidation headers and verification #14185

Conversation

chrisjrn commented Jan 18, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chrisjrn Jan 19, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chrisjrn commented Jan 19, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Eric-Arellano commented Jan 19, 2022

chrisjrn commented Jan 19, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chrisjrn commented Jan 19, 2022

Eric-Arellano left a comment

Choose a reason for hiding this comment

stuhood left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chrisjrn commented Jan 19, 2022

stuhood commented Jan 19, 2022

chrisjrn commented Jan 18, 2022 •

edited

chrisjrn Jan 19, 2022 •

edited