Git file install: support for file creation from Git repositories #2754

benfitzpatrick · 2024-01-17T13:03:15Z

Work in progress!

Aims to address #1419.

I'm not that happy with the configuration syntax, although I am happy with the three main elements of it as elements.

The mechanism itself uses filtering and either partial clones or proper sparse checkouts where the Git version allows.

wxtim · 2024-01-18T08:45:05Z

Let us know when you want us to have a review at this.

benfitzpatrick · 2024-01-23T13:32:30Z

Ready for review, not ready to go in IMO!

oliver-sanders

LGTM

.github/workflows/test.yml

metomi/rose/loc_handlers/git.py

oliver-sanders · 2024-02-26T10:29:34Z

Ready for review, not ready to go in IMO!

@benfitzpatrick, what's left to do?

benfitzpatrick · 2024-02-26T10:58:35Z

I'd like to do a bit of in-anger user-case testing if we feel it's close enough as-is? Shouldn't take too long.

benfitzpatrick · 2024-03-14T16:20:03Z

I've done some testing against real cases I could find in our workflows, found and fixed some issues in afddb2b.

I tested a case with 6 GitHub file installs in a single rose-app.conf - we were worried about hitting a rate limit. It worked fine and looked like these happen sequentially, at least from the verbose output? Happy to demo offline.

Otherwise I'm happy

oliver-sanders · 2024-03-15T16:19:22Z

looked like these happen sequentially

Curious, I'll check that's because GitHub is serving the requests sequentially rather than Rose failing to run them in parallel.

benfitzpatrick · 2024-04-10T13:47:08Z

Little bump

oliver-sanders · 2024-04-12T09:25:56Z

looked like these happen sequentially

Curious, I'll check that's because GitHub is serving the requests sequentially rather than Rose failing to run them in parallel.

After a bit of digging I'm happy that these operations are being run concurrently, resulting in their subprocesses being run in parallel.

To track subprocesses, try the following diff:

diff --git a/metomi/rose/popen.py b/metomi/rose/popen.py
index 8993fb93..8ec00c94 100644
--- a/metomi/rose/popen.py
+++ b/metomi/rose/popen.py
@@ -372,6 +372,7 @@ class RosePopener:
                 proc = Popen(args[0], **kwargs)
             else:
                 command = ' '.join(map(shlex.quote, args))
+                print(f'# {command}')  # process started
                 proc = await asyncio.create_subprocess_shell(command, **kwargs)
         except OSError as exc:
             if exc.filename is None and args:
@@ -392,7 +393,9 @@ class RosePopener:
         if isinstance(kwargs.get("stdin"), str):
             stdin = kwargs.get("stdin")
         stdout, stderr = await proc.communicate(stdin)
+        command = ' '.join(map(shlex.quote, args))
         await proc.wait()
+        print(f'$ {command}')  # process returned
         return proc.returncode, stdout, stderr
 
     async def run_ok_async(self, *args, **kwargs):

Using this diff with the following config:

[command]
default=true

[file:cylc/scheduler.py]
source=git:git@github.com:cylc/cylc-flow.git::cylc/flow/scheduler.py::master

[file:cylc/task_proxy.py]
source=git:https://github.com/cylc/cylc-flow.git::cylc/flow/task_proxy.py::master

[file:cylc/async_util.py]
source=git:https://github.com/cylc/cylc-flow.git::cylc/flow/async_util.py::master

I get this output showing up to three subprocesses running simultaneously:

# git --git-dir=/var/tmp/tmpqqkj_tvk/.git init
# git --git-dir=/var/tmp/tmpd4c71k8x/.git init
# git --git-dir=/var/tmp/tmp50f2vgwd/.git init
$ git --git-dir=/var/tmp/tmpqqkj_tvk/.git init
# git --git-dir=/var/tmp/tmpqqkj_tvk/.git remote add origin https://github.com/cylc/cylc-flow.git
$ git --git-dir=/var/tmp/tmp50f2vgwd/.git init
# git --git-dir=/var/tmp/tmp50f2vgwd/.git remote add origin https://github.com/cylc/cylc-flow.git
$ git --git-dir=/var/tmp/tmpd4c71k8x/.git init
# git --git-dir=/var/tmp/tmpd4c71k8x/.git remote add origin git@github.com:cylc/cylc-flow.git
$ git --git-dir=/var/tmp/tmpqqkj_tvk/.git remote add origin https://github.com/cylc/cylc-flow.git
# git --git-dir=/var/tmp/tmpqqkj_tvk/.git fetch --depth=1 origin 4b70574c9100e1cf830a226c254cb3159a558873
$ git --git-dir=/var/tmp/tmp50f2vgwd/.git remote add origin https://github.com/cylc/cylc-flow.git
# git --git-dir=/var/tmp/tmp50f2vgwd/.git fetch --depth=1 origin 4b70574c9100e1cf830a226c254cb3159a558873
$ git --git-dir=/var/tmp/tmpd4c71k8x/.git remote add origin git@github.com:cylc/cylc-flow.git
# git --git-dir=/var/tmp/tmpd4c71k8x/.git fetch --depth=1 origin 4b70574c9100e1cf830a226c254cb3159a558873
$ git --git-dir=/var/tmp/tmpqqkj_tvk/.git fetch --depth=1 origin 4b70574c9100e1cf830a226c254cb3159a558873
# git --git-dir=/var/tmp/tmpqqkj_tvk/.git --work-tree=/var/tmp/tmpqqkj_tvk checkout 4b70574c9100e1cf830a226c254cb3159a558873
$ git --git-dir=/var/tmp/tmp50f2vgwd/.git fetch --depth=1 origin 4b70574c9100e1cf830a226c254cb3159a558873
# git --git-dir=/var/tmp/tmp50f2vgwd/.git --work-tree=/var/tmp/tmp50f2vgwd checkout 4b70574c9100e1cf830a226c254cb3159a558873
$ git --git-dir=/var/tmp/tmp50f2vgwd/.git --work-tree=/var/tmp/tmp50f2vgwd checkout 4b70574c9100e1cf830a226c254cb3159a558873
# rsync -a '--exclude=.*' --timeout=1800 '--rsh=ssh -oBatchMode=yes -oStrictHostKeyChecking=no -oConnectTimeout=8' /var/tmp/tmp50f2vgwd/cylc/flow/async_util.py /var/tmp/tmph9lc16hm/f13f5dacdb0f632826821bf9278a10fc
$ git --git-dir=/var/tmp/tmpqqkj_tvk/.git --work-tree=/var/tmp/tmpqqkj_tvk checkout 4b70574c9100e1cf830a226c254cb3159a558873
# rsync -a '--exclude=.*' --timeout=1800 '--rsh=ssh -oBatchMode=yes -oStrictHostKeyChecking=no -oConnectTimeout=8' /var/tmp/tmpqqkj_tvk/cylc/flow/task_proxy.py /var/tmp/tmph9lc16hm/759c658589e37a9af232bfa8e03d01b0
$ rsync -a '--exclude=.*' --timeout=1800 '--rsh=ssh -oBatchMode=yes -oStrictHostKeyChecking=no -oConnectTimeout=8' /var/tmp/tmpqqkj_tvk/cylc/flow/task_proxy.py /var/tmp/tmph9lc16hm/759c658589e37a9af232bfa8e03d01b0
$ rsync -a '--exclude=.*' --timeout=1800 '--rsh=ssh -oBatchMode=yes -oStrictHostKeyChecking=no -oConnectTimeout=8' /var/tmp/tmp50f2vgwd/cylc/flow/async_util.py /var/tmp/tmph9lc16hm/f13f5dacdb0f632826821bf9278a10fc
$ git --git-dir=/var/tmp/tmpd4c71k8x/.git fetch --depth=1 origin 4b70574c9100e1cf830a226c254cb3159a558873
# git --git-dir=/var/tmp/tmpd4c71k8x/.git --work-tree=/var/tmp/tmpd4c71k8x checkout 4b70574c9100e1cf830a226c254cb3159a558873
$ git --git-dir=/var/tmp/tmpd4c71k8x/.git --work-tree=/var/tmp/tmpd4c71k8x checkout 4b70574c9100e1cf830a226c254cb3159a558873
# rsync -a '--exclude=.*' --timeout=1800 '--rsh=ssh -oBatchMode=yes -oStrictHostKeyChecking=no -oConnectTimeout=8' /var/tmp/tmpd4c71k8x/cylc/flow/scheduler.py /var/tmp/tmph9lc16hm/8a367e85378d4334d5133fa065d71116
$ rsync -a '--exclude=.*' --timeout=1800 '--rsh=ssh -oBatchMode=yes -oStrictHostKeyChecking=no -oConnectTimeout=8' /var/tmp/tmpd4c71k8x/cylc/flow/scheduler.py /var/tmp/tmph9lc16hm/8a367e85378d4334d5133fa065d71116

You can also test it like this:

diff --git a/metomi/rose/popen.py b/metomi/rose/popen.py
index 8993fb93..35bdbe27 100644
--- a/metomi/rose/popen.py
+++ b/metomi/rose/popen.py
@@ -371,7 +371,8 @@ class RosePopener:
             if kwargs.get("shell"):
                 proc = Popen(args[0], **kwargs)
             else:
-                command = ' '.join(map(shlex.quote, args))
+                # command = ' '.join(map(shlex.quote, args))
+                command = 'sleep 5'
                 proc = await asyncio.create_subprocess_shell(command, **kwargs)
         except OSError as exc:
             if exc.filename is None and args:

The time taken should increase linearly as the sleep is increased, but only see a small increase (parsing overheads) when more files are configured for installation.

sphinx/api/configuration/file-creation.rst

metomi/rose/loc_handlers/git.py

benfitzpatrick · 2024-04-15T16:06:22Z

Curious, I'll check that's because GitHub is serving the requests sequentially rather than Rose failing to run them in parallel.

After a bit of digging I'm happy that these operations are being run concurrently, resulting in their subprocesses being run in parallel.

Ah, I know why I thought they were sequential - the parsing is serial, the actual pulling can be concurrent. Sorry-thanks!

metomi/rose/loc_handlers/git.py

wxtim · 2024-06-03T10:56:41Z

metomi/rose/loc_handlers/git.py

+            if loc.loc_type == "tree":
+                dest += "/"
+            cmd = self.manager.popen.get_cmd("rsync", name, dest)
+            await self.manager.popen.run_ok_async(*cmd)


I'm getting a failure here:

[INFO] 2024-06-03T11:56:18+0100 rsync -a '--exclude=.*' --timeout=1800 '--rsh=ssh -oBatchMode=yes -oStrictHostKeyChecking=no -oConnectTimeout=8' /var/tmp/tmpzpxb2lpj/README.md /var/tmp/tmpjul8aid7/e99d80506f38bfee39893fd042f2f5d6 [FAIL] 2024-06-03T11:56:19+0100 rsync -a '--exclude=.*' --timeout=1800 '--rsh=ssh -oBatchMode=yes -oStrictHostKeyChecking=no -oConnectTimeout=8' /var/tmp/tmpzpxb2lpj/README.md /var/tmp/tmpjul8aid7/e99d80506f38bfee39893fd042f2f5d6 # return-code=3, stderr= [FAIL] 2024-06-03T11:56:19+0100 rsync: change_dir#3 "/var/tmp/tmpjul8aid7" failed: No such file or directory (2) [FAIL] 2024-06-03T11:56:19+0100 rsync error: errors selecting input/output files, dirs (code 3) at main.c(694) [Receiver=3.1.2]

(status update - we're finding it hard to independently reproduce this one, currently a mystery)

metomi/rose/loc_handlers/git.py

Co-authored-by: Oliver Sanders <oliver.sanders@metoffice.gov.uk>

oliver-sanders

Code review - LGTM, some small comments.

oliver-sanders · 2024-06-05T14:18:42Z

metomi/rose/config_processors/fileinstall.py

@@ -856,7 +857,7 @@ def parse(self, loc, conf_tree):
            # Scheme specified in the configuration.
            handler = self.get_handler(loc.scheme)
            if handler is None:
-                raise ValueError(loc.name)
+                raise ValueError(f"don't support scheme {loc.scheme}")


[minor] maybe a slightly cryptic error message for users to digest:

Suggested change

raise ValueError(f"don't support scheme {loc.scheme}")

raise ValueError(f"Rose doesn't support scheme {loc.scheme}")

oliver-sanders · 2024-06-05T14:19:04Z

metomi/rose/config_processors/fileinstall.py

@@ -865,7 +866,7 @@ def parse(self, loc, conf_tree):
                if handler is None:
                    handler = self.guess_handler(loc)
                if handler is None:
-                    raise ValueError(loc.name)
+                    raise ValueError(f"don't know how to process {loc.name}")


[minor] maybe a slightly cryptic error message for users to digest:

Suggested change

raise ValueError(f"don't know how to process {loc.name}")

raise ValueError(f"Rose doesn't know how to process {loc.name}")

oliver-sanders · 2024-06-05T14:43:14Z

metomi/rose/loc_handlers/git.py

+                # sparse-checkout available and suitable for this case.
+                await self.manager.popen.run_ok_async(
+                    "git", git_dir_opt, "sparse-checkout", "set", path,
+                    "--no-cone"


I'm guessing this is required, but worth noting the documented warnings:

https://git-scm.com/docs/git-sparse-checkout#_commands

oliver-sanders · 2024-06-05T14:44:05Z

metomi/rose/loc_handlers/git.py

+                "git", git_dir_opt, f"--work-tree={tmpdirname}", "checkout",
+                loc.key
+            )
+            name = tmpdirname + "/" + path


[minor] os.path.join would be nicer.

sphinx/api/configuration/file-creation.rst

oliver-sanders · 2024-06-05T14:51:04Z

sphinx/api/configuration/file-creation.rst

+         You should set ``git config uploadpack.allowFilter true`` and
+         ``git config uploadpack.allowAnySHA1InWant true`` on repositories
+         if you are setting them up to pull from.


Maybe worth explaining that this only applies to repos within your management (e.g. local repos).

E.G. this doesn't apply to a GitHub hosted repository (I think)?

oliver-sanders

🚀

wxtim · 2024-06-06T12:10:15Z

I've opened #2784 - it looks like the bug I found is unrelated to this work.

Co-authored-by: Oliver Sanders <oliver.sanders@metoffice.gov.uk>

metomi/rose/loc_handlers/git.py

Co-authored-by: Ronnie Dutta <61982285+MetRonnie@users.noreply.github.com>

oliver-sanders · 2024-06-13T12:50:14Z

Whilst working on #2785, I've noticed that "cannot connect to" type errors appear to surface with the following message:

ConfigProcessError: file:README.md=source=git:https@github.com:metomi/rose::README.md::master: ls-remote: could not find ref 'master' in 'git@github.com:metomi/rose'

Where the underlying cause here is that GitHub SSH access has not been configured.

benfitzpatrick · 2024-06-14T08:52:57Z

I could do a "or remote not contactable?" on the end of the error message?

oliver-sanders · 2024-06-14T09:42:33Z

If it's possible to differentiate between these cases, great, but if not that'll be fine.

metomi/rose/loc_handlers/git.py

git: differentiate between no such branch and no such repo

oliver-sanders · 2024-06-18T09:27:58Z

Waiting for tests to pass again, then ready to merge.

oliver-sanders · 2024-06-18T10:02:29Z

Good news, the tests pass.

Bad news, if some of the tests skip, then the test fails, e.g:

diff --git a/t/rose-app-run/28-git.t b/t/rose-app-run/28-git.t
index 1b4eaaa90..92a411611 100644
--- a/t/rose-app-run/28-git.t
+++ b/t/rose-app-run/28-git.t
@@ -72,12 +72,12 @@ remote_locations=("$HOSTNAME:$TEST_DIR/hellorepo/" "http://localhost:$GIT_WS_POR
for i in 0 1 2; do
     remote_mode="${remote_test_modes[$i]}"
     remote="${remote_locations[$i]}"
-    if [[ "$remote_mode" == "ssh" ]] && ! ssh -n -q -oBatchMode=yes $HOSTNAME true 1>'/dev/null' 2>/dev/null; then
+    if true; then
         skip 14 "cannot ssh to localhost $HOSTNAME"
         echo "Skip $remote" >/dev/tty        
         continue
     fi
-    if [[ "$remote_mode" == "http" ]] && ! curl --head --silent --fail $remote >/dev/null 2>&1; then
+    if true; then
         skip 14 "failed to launch http on localhost"
         echo "Skip $remote" >/dev/tty        
         continue

:(

[FAIL] ../config: rose-app.conf not found.
Dubious, test returned 1 (wstat 256, 0x100)
Failed 15/57 subtests 
	(less 42 skipped subtests: 0 okay)
Test Summary Report
-------------------
t/rose-app-run/28-git.t (Wstat: 256 Tests: 42 Failed: 0)
  Non-zero exit status: 1
  Parse errors: Bad plan.  You planned 57 tests but ran 42.

Not a release blocker. Suggest bumping this to future work.

~~Opened: #2789~~

oliver-sanders · 2024-06-18T10:04:12Z

@MetRonnie, could you check the last couple of commits and merge if you're happy with the tests being pushed to a bugfix release milestone.

oliver-sanders · 2024-06-18T10:28:30Z

Brill, that's fixed it (will close issue). Tested with all possible skip combinations, test passed each time.

oliver-sanders · 2024-06-18T10:40:10Z

(Mac OS tests failing due to gh-actions DNS issues, safe to ignore)

MetRonnie

merge if you're happy with the tests being pushed to a bugfix release milestone.

It's a minor release rather than a bugfix release though?

oliver-sanders · 2024-06-18T10:46:08Z

(the test fixes would have been on a bugfix release milestone, however, there is no need for a post-fix, Ben has sorted it on this PR)

benfitzpatrick force-pushed the 1419.git_file_creation_loc_handler branch from 28e76e2 to 689784e Compare January 17, 2024 14:26

oliver-sanders added this to the 2.3.0 milestone Jan 18, 2024

oliver-sanders assigned benfitzpatrick Jan 18, 2024

benfitzpatrick marked this pull request as ready for review January 23, 2024 13:32

oliver-sanders reviewed Feb 5, 2024

View reviewed changes

.github/workflows/test.yml Outdated Show resolved Hide resolved

metomi/rose/loc_handlers/git.py Outdated Show resolved Hide resolved

MetRonnie self-requested a review February 23, 2024 11:46

MetRonnie reviewed Feb 23, 2024

View reviewed changes

metomi/rose/loc_handlers/git.py Outdated Show resolved Hide resolved

oliver-sanders requested review from MetRonnie and oliver-sanders March 15, 2024 16:17

oliver-sanders reviewed Apr 12, 2024

View reviewed changes

sphinx/api/configuration/file-creation.rst Outdated Show resolved Hide resolved

metomi/rose/loc_handlers/git.py Outdated Show resolved Hide resolved

metomi/rose/loc_handlers/git.py Show resolved Hide resolved

benfitzpatrick commented Apr 24, 2024

View reviewed changes

metomi/rose/loc_handlers/git.py Show resolved Hide resolved

wxtim reviewed Jun 3, 2024

View reviewed changes

benfitzpatrick added 3 commits June 5, 2024 11:08

The start

a044d52

Remove unnecessary/ineffective commands thanks to review

64a17e5

Minimal parsing, inferred loc type, fewer commands and less bandwidth

5a16399

benfitzpatrick force-pushed the 1419.git_file_creation_loc_handler branch from ad11e99 to 5a16399 Compare June 5, 2024 10:08

oliver-sanders reviewed Jun 5, 2024

View reviewed changes

metomi/rose/loc_handlers/git.py Outdated Show resolved Hide resolved

benfitzpatrick and others added 2 commits June 5, 2024 11:38

Update metomi/rose/loc_handlers/git.py

e54d504

Co-authored-by: Oliver Sanders <oliver.sanders@metoffice.gov.uk>

More comments

b4f653f

oliver-sanders reviewed Jun 5, 2024

View reviewed changes

oliver-sanders approved these changes Jun 5, 2024

View reviewed changes

wxtim mentioned this pull request Jun 6, 2024

Failure to install from subversion #2784

Closed

wxtim self-requested a review June 6, 2024 12:10

oliver-sanders mentioned this pull request Jun 6, 2024

job-runner: fix async interaction #2785

Merged

Update sphinx/api/configuration/file-creation.rst

0390b49

Co-authored-by: Oliver Sanders <oliver.sanders@metoffice.gov.uk>

MetRonnie approved these changes Jun 11, 2024

View reviewed changes

metomi/rose/loc_handlers/git.py Outdated Show resolved Hide resolved

metomi/rose/loc_handlers/git.py Outdated Show resolved Hide resolved

metomi/rose/loc_handlers/git.py Outdated Show resolved Hide resolved

metomi/rose/loc_handlers/git.py Show resolved Hide resolved

benfitzpatrick and others added 2 commits June 13, 2024 08:48

Update metomi/rose/loc_handlers/git.py

5bc2f97

Co-authored-by: Ronnie Dutta <61982285+MetRonnie@users.noreply.github.com>

metomi#1419: address @MetRonnie's feedback

fab495c

oliver-sanders reviewed Jun 17, 2024

View reviewed changes

metomi/rose/loc_handlers/git.py Show resolved Hide resolved

oliver-sanders and others added 3 commits June 17, 2024 15:16

git: differentiate between no such branch and no such repo

0c5af7f

Update metomi/rose/loc_handlers/git.py

83623bc

Merge pull request #18 from oliver-sanders/git-no-such-repo

ec5e917

git: differentiate between no such branch and no such repo

MetRonnie approved these changes Jun 18, 2024

View reviewed changes

Fix mistake, tests pass

4851eda

oliver-sanders mentioned this pull request Jun 18, 2024

loc_handlers git: fix potentially flaky test t/rose-app-run/28-git.t #2789

Closed

oliver-sanders requested a review from MetRonnie June 18, 2024 10:04

Fix test when all previous ones skipped

460bbf7

MetRonnie approved these changes Jun 18, 2024

View reviewed changes

oliver-sanders merged commit 3bd74d5 into metomi:master Jun 18, 2024
12 of 13 checks passed

benfitzpatrick deleted the 1419.git_file_creation_loc_handler branch June 25, 2024 10:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Git file install: support for file creation from Git repositories #2754

Git file install: support for file creation from Git repositories #2754

benfitzpatrick commented Jan 17, 2024

wxtim commented Jan 18, 2024

benfitzpatrick commented Jan 23, 2024

oliver-sanders left a comment

oliver-sanders commented Feb 26, 2024

benfitzpatrick commented Feb 26, 2024

benfitzpatrick commented Mar 14, 2024

oliver-sanders commented Mar 15, 2024

benfitzpatrick commented Apr 10, 2024

oliver-sanders commented Apr 12, 2024

benfitzpatrick commented Apr 15, 2024

wxtim Jun 3, 2024

benfitzpatrick Jun 5, 2024

oliver-sanders left a comment

oliver-sanders Jun 5, 2024

oliver-sanders Jun 5, 2024

oliver-sanders Jun 5, 2024

oliver-sanders Jun 5, 2024

oliver-sanders Jun 5, 2024

oliver-sanders left a comment

wxtim commented Jun 6, 2024

oliver-sanders commented Jun 13, 2024

benfitzpatrick commented Jun 14, 2024

oliver-sanders commented Jun 14, 2024

oliver-sanders commented Jun 18, 2024

oliver-sanders commented Jun 18, 2024 •

edited

Loading

oliver-sanders commented Jun 18, 2024

oliver-sanders commented Jun 18, 2024

oliver-sanders commented Jun 18, 2024

MetRonnie left a comment

oliver-sanders commented Jun 18, 2024

	raise ValueError(f"don't support scheme {loc.scheme}")
	raise ValueError(f"Rose doesn't support scheme {loc.scheme}")

	raise ValueError(f"don't know how to process {loc.name}")
	raise ValueError(f"Rose doesn't know how to process {loc.name}")

Git file install: support for file creation from Git repositories #2754

Git file install: support for file creation from Git repositories #2754

Conversation

benfitzpatrick commented Jan 17, 2024

wxtim commented Jan 18, 2024

benfitzpatrick commented Jan 23, 2024

oliver-sanders left a comment

Choose a reason for hiding this comment

oliver-sanders commented Feb 26, 2024

benfitzpatrick commented Feb 26, 2024

benfitzpatrick commented Mar 14, 2024

oliver-sanders commented Mar 15, 2024

benfitzpatrick commented Apr 10, 2024

oliver-sanders commented Apr 12, 2024

benfitzpatrick commented Apr 15, 2024

wxtim Jun 3, 2024

Choose a reason for hiding this comment

benfitzpatrick Jun 5, 2024

Choose a reason for hiding this comment

oliver-sanders left a comment

Choose a reason for hiding this comment

oliver-sanders Jun 5, 2024

Choose a reason for hiding this comment

oliver-sanders Jun 5, 2024

Choose a reason for hiding this comment

oliver-sanders Jun 5, 2024

Choose a reason for hiding this comment

oliver-sanders Jun 5, 2024

Choose a reason for hiding this comment

oliver-sanders Jun 5, 2024

Choose a reason for hiding this comment

oliver-sanders left a comment

Choose a reason for hiding this comment

wxtim commented Jun 6, 2024

oliver-sanders commented Jun 13, 2024

benfitzpatrick commented Jun 14, 2024

oliver-sanders commented Jun 14, 2024

oliver-sanders commented Jun 18, 2024

oliver-sanders commented Jun 18, 2024 • edited Loading

oliver-sanders commented Jun 18, 2024

oliver-sanders commented Jun 18, 2024

oliver-sanders commented Jun 18, 2024

MetRonnie left a comment

Choose a reason for hiding this comment

oliver-sanders commented Jun 18, 2024

oliver-sanders commented Jun 18, 2024 •

edited

Loading