Add support to build GE for Windows on Arm64 (WoA) #10649

chirontt · 2023-01-15T23:30:31Z

Resolves #10674

With this PR, here are the commands to build GE on a Windows on Arm64 (WoA) platform:

build the portable GE with the command:

> dotnet build

build the shell extensions for WoA:

> dotnet build .\scripts\native_arm64.proj

The above command produces a GitExtensionsShellEx64.dll file for WoA, but GE also requires a GitExtensionsShellEx32.dll file to exist, so just make a copy of GitExtensionsShellEx64.dll to be GitExtensionsShellEx32.dll as well:

> copy artifacts\Debug\bin\GitExtensionsShellEx\GitExtensionsShellEx64.dll artifacts\Debug\bin\GitExtensionsShellEx\GitExtensionsShellEx32.dll

package the GE distribution:

> dotnet publish

The last command would fail to produce a .msi distribution, due to it using the WiX library which is currently incompatible with the WoA platform, but a portable GE zip distribution for WoA will be successfully produced in the folder artifacts\Debug\publish

Proposed changes

Build for Windows on Arm64 platform

Test methodology

Mostly manual tests, done on normal daily usage of the app.

Executing the automated tests, with the dotnet test command, mostly fail, with error about testhost.dll file:

Testhost process for source(s) 'C:\dev\dotnet\gitextensions\artifacts\Debug\bin\tests\UnitTests\GitUI.Tests\net6.0-windows\GitUI.Tests.dll' exited with error: Unhandled exception. System.BadImageFormatException: Could not load file or assembly 'C:\dev\dotnet\gitextensions\artifacts\Debug\bin\tests\UnitTests\GitUI.Tests\net6.0-windows\testhost.dll'. An attempt was made to load a program with an incorrect format.
File name: 'C:\dev\dotnet\gitextensions\artifacts\Debug\bin\tests\UnitTests\GitUI.Tests\net6.0-windows\testhost.dll'
. Please check the diagnostic logs for more information.

I guess testhost.dll is an x86 library, so it fails to load in an Arm64 GE process.

Test environment(s)

Git for Windows 2.39.0 ARM64
Windows 11 on Arm64
No UI scaling

Merge strategy

I agree that the maintainer squash merge this PR (if the commit message is clear).

✒️ I contribute this code under The Developer Certificate of Origin.

RussKie · 2023-01-16T00:22:16Z

Related: #10288

RussKie

🎉

The installer will need to be changed as well, see

gitextensions/Setup/Product.wxs

Lines 231 to 249 in 1ee9b75

    
           <!-- Shell extensions --> 
        
           <?define ShellExCLSID = "{3C16B20A-BA16-4156-916F-0A375ECFFE24}"?> 
        
           <Component Id="GitExtensionsShellEx32.dll" Guid="*"> 
        
             <File Name="GitExtensionsShellEx32.dll" Source="$(var.ArtifactsBinPath)\GitExtensionsShellEx\GitExtensionsShellEx32.dll" KeyPath="yes" /> 
        
             <?foreach ShellExFileName in GitExtensionsShellEx32.dll?> 
        
             <?include RegisterShellExtension.wxi?> 
        
             <?endforeach ?> 
        
           </Component> 
        
           <Component Id="GitExtensionsShellEx64.dll" Guid="*" Win64="yes"> 
        
             <Condition>VersionNT64</Condition> 
        
             <File Name="GitExtensionsShellEx64.dll" Source="$(var.ArtifactsBinPath)\GitExtensionsShellEx\GitExtensionsShellEx64.dll" KeyPath="yes" /> 
        
             <?foreach ShellExFileName in GitExtensionsShellEx64.dll?> 
        
             <?include RegisterShellExtension.wxi?> 
        
             <?endforeach ?> 
        
           </Component> 
        
           <!-- End shell extensions --> 
        
           <Component Id="GitExtSshAskPass.exe" Guid="*"> 
        
             <File Source="$(var.ArtifactsBinPath)\GitExtSshAskPass\GitExtSshAskPass.exe" /> 
        
           </Component>

Also, ConEmu doesn't work on Arm64, so some code paths will need to be conditioned:

RussKie · 2023-01-16T00:19:14Z

scripts/native_arm64.proj

+      <_ProjectArgsARM64> /p:Configuration=$(Configuration) /p:Platform=ARM64</_ProjectArgsARM64>
+    </PropertyGroup>
+
+    <!-- Build GitExtSshAskPass project, ARM64 -->
+    <Exec
+        Command="msbuild.exe $(_GitExtSshAskPassPath) $(_ProjectArgsARM64) /p:OutDir=$(_OutputPath)GitExtSshAskPass\ /bl:$(_ArtifactsLogDir)\GitExtSshAskPass.binlog"
+        WorkingDirectory="$(_MSBuildCurrentPath)"
+        EchoOff="true"
+        ConsoleToMsBuild="true"
+        StandardOutputImportance="High">
+    </Exec>
+    <!-- Build GitExtensionsShellEx project, ARM64 -->
+    <Exec
+        Command="msbuild.exe $(_GitExtensionsShellExPath) $(_ProjectArgsARM64) /p:OutDir=$(_OutputPath)GitExtensionsShellEx\ /bl:$(_ArtifactsLogDir)\GitExtensionsShellEx64.binlog"
+        WorkingDirectory="$(_MSBuildCurrentPath)"
+        EchoOff="true"
+        ConsoleToMsBuild="true"
+        StandardOutputImportance="High">
+    </Exec>


Is there a reason we can't do this in native.proj?

As I mentioned before, I can't produce the .msi installer for WoA, due to the WiX incompatibility problem, so I haven't got a chance to see what's needed for it. I've tried to recompile the WiX library for WoA, but it's so old that it throws so many errors (it's pre-net6 program, and .NET SDK in WoA only supports net6+) that I just give it up.

ConEmu problem in WoA was reported almost 5 years ago, and no progress since. I guess we need to wait a couple of more years still. So for now, I just disable ConEmu in my GE settings in WoA. Actually I want to even hide the Console tab so that I don't accidentally click it and produce a zombie process in the background:

Any idea how to hide it from being displayed on screen?

As to the new native_arm64.proj file:

Is there a reason we can't do this in native.proj?

Nope, just combining them together would cause compile problems on either platforms: they would share objs between different platforms (on WoA) which is a no-no, the x64 compiler wouldn't know how to compile the ARM64 platform which necessitates some needed conditionals to go different compile path for each platform, etc., that I reckon more research are needed to make it work. For now, using a separate proj file for Arm64 is the simplest solution I can think of.

Any idea how to hide it from being displayed on screen?

Look at the 'Build report' tab logic. I think we don't display it always. Maybe it's just a matter of not adding it to the tab collection.

Any idea how to hide it from being displayed on screen?

Make sure these options are always false (and settings disabled) for WoA.

@gerhardol Yes, the following setting will hide the Console tab:

GitExtensions/GitExtensions.csproj

RussKie · 2023-01-17T06:13:44Z

I wouldn't rely on a setting, instead I think we should conditionally exclude the implementation either with `#if ARM64` or `if Platform == "arm64"` (pseudo code).

chirontt · 2023-01-23T01:29:10Z

FYI: I've made my unofficial build of GE v4.0.2 for Windows Arm64 available on my fork.

scripts/native.proj

RussKie

Looks good 👍

Please address the outstanding comments and rebase.

What would be the steps to build a portable version on AppVeyor (since we can't produce an msi)?

Also, we'll need to add build instructions for the ARM64 builds. If you can provide a text, we'll have it added to https://github.com/gitextensions/gitextensions/wiki/How-To%3A-build-instructions.

GitExtensions/GitExtensions.csproj

RussKie · 2023-03-11T11:16:24Z

scripts/native.proj

@@ -43,7 +43,7 @@



Suggest renaming this file to native.amd.proj

OK, and as a result, the reference to it in appveyor.yml is updated too. So this renaming of native.proj makes appveyor.yml become exclusively X64 (and thus should be renamed to appveyor.amd.yml to be explicitly clear?)

There are few other places where native.proj is used: https://github.com/gitextensions/gitextensions/search?q=native.proj

So this renaming of native.proj makes appveyor.yml become exclusively X64 (and thus should be renamed to appveyor.amd.yml to be explicitly clear?)

I don't follow, sorry. We continue to build x86 as well, hence the suggestion to rename to native.amd.proj and not native.amd64.proj.

There are few other places where native.proj is used: https://github.com/gitextensions/gitextensions/search?q=native.proj

OK, due to these reasons, I've decided to revert to the original native.proj file name, and have merged the native.arm64.proj to it so that the native.proj will now support both Intel/AMD and ARM64 builds via the usage of an extra property. And that means the original appveyor.yml file won't be changed after all, and such it will continue building the x86/x64 products as originally intended.

I don't have any experience with AppVeyor so I can't do much about how to change appveyor.yml to support the extra ARM64 build.

With my latest changes, to build the x86/x64 natives would use the same command as originally intended:

dotnet build .\scripts\native.proj

while to build the ARM64 natives would require setting an extra Arm64Build flag:

dotnet build .\scripts\native.proj /p:Arm64Build=true

You could duplicate the block at line 73-77:

- ps: | # build .NET $cmd = "dotnet build -c Release /p:Arm64Build=true --verbosity q --nologo /bl:.\artifacts\log\build.binlog $buildArgs" Invoke-Expression $cmd if ($LastExitCode -ne 0) { $host.SetShouldExit($LastExitCode) }

Though it will only come into affect after merging the PR.

That's a good idea. Few things to note:

since we're build amd and arm we should distingush between the binlogs, hence this one should be named something like build.arm64.binlog;

we'll need to distinguish the zips as well, i.e., name the zip something like "GitExtensions-portable.arm64.xyz0123.zip".

You could duplicate the block at line 73-77:

No, it wouldn't work. The command:

dotnet build .\scripts\native.proj /p:Arm64Build=true

is meant to be executed on a WoA machine (or VM); it won't work if running on x64 platform or VM such as the one used by AppVeyor (I presume.) The reason is that the MSVC compiler for x86/x64 doesn't know how to generate Arm64 binaries.

This is why I suggested a new appveyor.arm64.yml to produce the Arm64 GE products, while keeping the existing appveyor.yml exclusively for x86/x64 GE products. But appveyor.arm64.yml would only work if AppVeyor could run on a WoA platform, and this is a big IF that I'm not sure about.

Makes sense.

RussKie · 2023-03-11T11:16:39Z

scripts/native_arm64.proj

@@ -0,0 +1,63 @@
+<Project DefaultTargets="Build">


...and this to native.arm64.proj

OK.

I think to handle Arm64 build, a new appveyor.arm64.yml file is needed (almost a duplicate of the current appveyor.yml contents.) But because the dotnet publish command will fail in WoA which would cause AppVeyor build process to also fail. Is there a way to change the dotnet publish command to produce the portable zip distribution only when building for Arm64?

I don't think it's possible to run multiple build pipelines on a free account. Aren't we able to build both of distros in one pipeline?

Is there a way to change the dotnet publish command to produce the portable zip distribution only when building for Arm64?

The publishing targets are here: https://github.com/gitextensions/gitextensions/blob/master/GitExtensions/Project.Publish.targets#L271. I don't think it should be too difficult to condition CreateMsi target based on a MSBuild property.

I don't think it's possible to run multiple build pipelines on a free account. Aren't we able to build both of distros in one pipeline?

I'm out of my depth here, so I can't comment any further.

The publishing targets are here: https://github.com/gitextensions/gitextensions/blob/master/GitExtensions/Project.Publish.targets#L271. I don't think it should be too difficult to condition CreateMsi target based on a MSBuild property.

Yes, it's trivial to add a condition to the CreateMsi target based on a property, but deciding on which property to use is tricky. I've tried various existing properties like ProcessorArchitecture, RuntimeIdentifier, Platform by adding extra options such as -a ARM64 or -r win-arm64 or --use-current-runtime to the dotnet publish command, but they have various side-effects in the resulting package. Like, using the -r win-arm64 or --use-current-runtime on WoA would produce a package for Arm64 having similar problems as mentioned in #9990.

Eventually I decided to define my own property Arm64Build, so that to produce the x86/x64 distributions the original command would still work as before:

dotnet publish

while to produce Arm64 package would require setting the Arm64Build flag so that it would skip the CreateMsi step:

dotnet publish /p:Arm64Build=true

In summary, here are the complete (unchanged) commands to build GE for x86/x64 platform:

dotnet build dotnet build .\scripts\native.proj dotnet publish

while to build GE for WoA:

dotnet build dotnet build .\scripts\native.proj /p:Arm64Build=true dotnet publish /p:Arm64Build=true

Ideally, the first set of commands should have worked for both situations, without defining the new Arm64Build property, but I haven't found any existing MSBuild property up to the task. Any help/suggestion is quite welcome.

RussKie

👍

RussKie · 2023-03-19T23:02:46Z

scripts/native.proj

    </PropertyGroup>

    <!-- Build GitExtSshAskPass project, x86 -->
    <Exec
+        Condition="!$(Arm64Build)"


This is error-prone as we don't control the value. We should do this instead:

Suggested change

Condition="!$(Arm64Build)"

Condition=" '$(Arm64Build)' != 'true' "

Ditto for all other similar instances.

I disagree. Arm64Build is meant to be a boolean flag, with true value being the only value having significant effect; any other value is simply not true i.e. false. I'm just following MSBuild docs here. The fact that Arm64Build is implemented as a string is just a shortcoming of MSBuild, so making it explicit like '$(Arm64Build)' != 'true' is clunky and unnecessary, in my opinion, as I want to hide its silly string implementation!

dotnet publish /p:Arm64Build=yes will cause this to fail.
MSBuild doesn't have strong typing, and because of that we should be defensive in our implementations. This is how comparisons are done throughout the .NET SDK.

dotnet publish /p:Arm64Build=yes will cause this to fail.

Yes, and it should fail for Arm64 build. As I mentioned before, Arm64Build is a boolean flag with true having significant meaning, any other values are false.

MSBuild properties aren't strongly typed. So they have to be compared in a safe way.

RussKie · 2023-03-19T23:07:53Z

GitExtensions/Project.Publish.targets

@@ -268,7 +268,10 @@
    Creates an MSI.
    ============================================================
    -->
-  <Target Name="CreateMsi" AfterTargets="CreatePortable">
+  <PropertyGroup>
+    <Arm64Build>false</Arm64Build>


Propose a rename to

Suggested change

<Arm64Build>false</Arm64Build>

<IsArm64>false</IsArm64>

I think a better name for the flag would be:

<IsArm64Build>false</IsArm64Build>

thus making it more meaningfully a boolean property.

Works for me.

RussKie · 2023-03-19T23:11:48Z

scripts/native.proj

@@ -43,7 +43,7 @@



That's a good idea. Few things to note:

since we're build amd and arm we should distingush between the binlogs, hence this one should be named something like build.arm64.binlog;

we'll need to distinguish the zips as well, i.e., name the zip something like "GitExtensions-portable.arm64.xyz0123.zip".

RussKie

As per the earlier comments

RussKie · 2023-04-21T02:21:40Z

@AArnott if you could take this for a spin it'd be great. Also, a review is welcome ;)

ghost · 2023-05-05T03:00:46Z

This pull request has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 14 days. It will be closed if no further activity occurs.

AArnott · 2023-05-17T04:39:36Z

@RussKie sorry I'm just now getting to this. How do I test it? Are there binaries I can download from somewhere?

RussKie · 2023-05-18T02:21:28Z

Sorry, no binaries, we don't have arm64 infra to build. If you have an arm64 device you could pull this PR and see if you can build an arm64-native app. I think these commands should work: dotnet build .\scripts\native.proj /p:Arm64Build=true dotnet publish /p:Arm64Build=true

AArnott · 2023-05-18T12:20:23Z

Thanks. Building was as easy as you say. But the result was an .exe that still ran as x64.

Its headers (from ILSpy) might suggest a reason:

// C:\Users\andre\source\repos\gitextensions\artifacts\Debug\publish\GitExtensions\GitExtensions.exe
// DotnetRuntimeBootstrapper.AppHost.Gui, Version=2.3.1.0, Culture=neutral, PublicKeyToken=null
// Global type: <Module>
// Entry point: DotnetRuntimeBootstrapper.AppHost.Gui.Program.Main
// Architecture: AnyCPU (64-bit preferred)
// Runtime: v2.0.50727
// This assembly was compiled using the /deterministic option.
// Hash algorithm: SHA1

It's AnyCPU. And it appears to target the .NET Framework 2.0 runtime, which is weird because GitExtensions.dll targets .NET 6. So maybe you're playing some weird tricks anyway. But .NET Framework AnyCPU .exes run as x64 even on an arm64 machine. You have to target arm64 specifically in an .exe to get an arm64 process.

RussKie · 2023-05-18T12:51:21Z

Please make sure to clone the submodules. Also the branch is out of date, we no longer use dotnetbootstrapper.

AArnott · 2023-05-18T12:54:48Z

Yes, I had to clone the submodules for the dotnet publish step to work.
So I guess you're saying I need to merge your source and target branches before building?

vbjay · 2023-05-18T13:01:33Z

You can rebase to keep history clean.

RussKie · 2023-05-18T13:02:23Z

It's a good idea to rebase this branch on top of the latest master. IIRC dotnetbootstrapper was a .NET Framework host that would download the required .NET Desktop runtime, if missing, and then start the app. But we solved the bootstrap problem in another way and removed the dependency.

RussKie · 2023-05-18T13:03:22Z

You're welcome to ping me in teams, if you still have troubles ;)

AArnott · 2023-05-18T13:05:24Z

History doesn't matter in this case because I'm just testing a PR. I'm not pushing anything.
And anyway I generally disagree with rebasing (unless I'm cleaning history). This PR has 4 commits, and if there were merge conflicts, rebasing could make me resolve conflicts 4 times, whereas merging only presents me with one set of conflicts.

Regardless.... the test worked. I have an arm64 gitextensions.exe process and I poked around the window and switched views and it seemed to work. Thank you.

Sorry, no binaries, we don't have arm64 infra to build.

You shouldn't need arm64 infra to build arm64 binaries. Ever hear of cross-compiling?
And how are you going to actually build release binaries for arm64 if your PR/CI can't build arm64 binaries?

vbjay · 2023-05-18T23:11:57Z

Fyi. https://git-scm.com/book/en/v2/Git-Tools-Rerere

You can prep with a merge and solve merge conflicts then reset and rebase.

AArnott · 2023-05-19T01:49:22Z

I love git rerere, but as each rebased commit can introduce different conflicts, I don't think it is a guarantee that you only have to resolve conflicts once during a rebase.
And I just don't see the value of rebasing many commits when a merge commit will do, and ensure a more reconstructible history. Part of this is a philosophical difference that probably doesn't need to happen on this github issue.

I'd love to hear how we can get the arm64 build shipping though.

RussKie · 2023-05-21T06:53:26Z

This PR has 4 commits, and if there were merge conflicts, rebasing could make me resolve conflicts 4 times, whereas merging only presents me with one set of conflicts.

That's not necessarily true. But in this case, as you said, it doesn't matter.

You shouldn't need arm64 infra to build arm64 binaries. Ever hear of cross-compiling?
And how are you going to actually build release binaries for arm64 if your PR/CI can't build arm64 binaries?

I think this is how dotnet/windowsdesktop does it, but I never actually looked into it.
We have no immediate plans to build or release arm64 binaries, I viewed this work as an enabled for folks on arm64 to be able to build it locally. I don't think anyone from the Core team (which is like 3.5 individuals, me being a 0.5) has any spare cycles to work on this initiative. If we get a contributor willing and able to help to make it build, we'll happily take it :)

AArnott · 2023-05-21T13:54:33Z

I viewed this work as an enabled for folks on arm64 to be able to build it locally

Well thank you for that!

If we get a contributor willing and able to help to make it build,

I'll be happy to look into updating your pipeline to include an arm64 build.
Any chance of this PR merging so that I have something to build?

To build GE for WoA, run the command: dotnet build /p:Arm64Build=true To package GE distribution for WoA, run the commands: dotnet build .\scripts\native.proj /p:Arm64Build=true dotnet publish /p:Arm64Build=true The last command would fail to produce a .msi distribution, due to it using the WiX library which is currently incompatible with the WoA platform, but a portable GE zip distribution for WoA is successfully produced in the folder: artifacts\Debug\publish

RussKie · 2023-05-22T04:12:05Z

Thank you @chirontt for the original contribution and @AArnott for the verification.

ghost assigned chirontt Jan 15, 2023

RussKie reviewed Jan 16, 2023

View reviewed changes

chirontt force-pushed the support_windows_on_arm64 branch from 2f2a39a to ae3bd9b Compare January 16, 2023 01:17

chirontt marked this pull request as ready for review January 23, 2023 01:29

gerhardol mentioned this pull request Jan 29, 2023

Version 4.0.2 doesn't run on Windows 11 ARM64 #10674

Closed

RussKie reviewed Mar 11, 2023

View reviewed changes

scripts/native.proj Show resolved Hide resolved

RussKie reviewed Mar 11, 2023

View reviewed changes

RussKie added the 📭 needs: author feedback More info/confirmation awaited from OP; issues typically get closed after 30 days of inactivity label Mar 11, 2023

ghost removed the 📭 needs: author feedback More info/confirmation awaited from OP; issues typically get closed after 30 days of inactivity label Mar 13, 2023

RussKie added the 📭 needs: author feedback More info/confirmation awaited from OP; issues typically get closed after 30 days of inactivity label Mar 14, 2023

ghost removed the 📭 needs: author feedback More info/confirmation awaited from OP; issues typically get closed after 30 days of inactivity label Mar 19, 2023

RussKie reviewed Mar 19, 2023

View reviewed changes

RussKie added the 📭 needs: author feedback More info/confirmation awaited from OP; issues typically get closed after 30 days of inactivity label Mar 26, 2023

ghost removed the 📭 needs: author feedback More info/confirmation awaited from OP; issues typically get closed after 30 days of inactivity label Apr 5, 2023

RussKie requested changes Apr 10, 2023

View reviewed changes

ghost added the 📭 needs: author feedback More info/confirmation awaited from OP; issues typically get closed after 30 days of inactivity label Apr 10, 2023

ghost added the 💤 status: no recent activity The issue is stale label May 5, 2023

ghost closed this May 12, 2023

ghost added the status: abandoned label May 12, 2023

ghost removed the 💤 status: no recent activity The issue is stale label May 17, 2023

RussKie reopened this May 22, 2023

RussKie force-pushed the support_windows_on_arm64 branch from e173f2c to a9fcf72 Compare May 22, 2023 02:30

RussKie force-pushed the support_windows_on_arm64 branch from a9fcf72 to f98b438 Compare May 22, 2023 02:32

RussKie approved these changes May 22, 2023

View reviewed changes

RussKie removed 📭 needs: author feedback More info/confirmation awaited from OP; issues typically get closed after 30 days of inactivity status: abandoned labels May 22, 2023

RussKie merged commit a504b15 into gitextensions:master May 22, 2023
4 checks passed

ghost added this to the vNext milestone May 22, 2023

AArnott mentioned this pull request Sep 14, 2023

Build arm64 in CI #11210

Draft

	<!-- Shell extensions -->
	<?define ShellExCLSID = "{3C16B20A-BA16-4156-916F-0A375ECFFE24}"?>
	<Component Id="GitExtensionsShellEx32.dll" Guid="*">
	<File Name="GitExtensionsShellEx32.dll" Source="$(var.ArtifactsBinPath)\GitExtensionsShellEx\GitExtensionsShellEx32.dll" KeyPath="yes" />
	<?foreach ShellExFileName in GitExtensionsShellEx32.dll?>
	<?include RegisterShellExtension.wxi?>
	<?endforeach ?>
	</Component>
	<Component Id="GitExtensionsShellEx64.dll" Guid="*" Win64="yes">
	<Condition>VersionNT64</Condition>
	<File Name="GitExtensionsShellEx64.dll" Source="$(var.ArtifactsBinPath)\GitExtensionsShellEx\GitExtensionsShellEx64.dll" KeyPath="yes" />
	<?foreach ShellExFileName in GitExtensionsShellEx64.dll?>
	<?include RegisterShellExtension.wxi?>
	<?endforeach ?>
	</Component>
	<!-- End shell extensions -->
	<Component Id="GitExtSshAskPass.exe" Guid="*">
	<File Source="$(var.ArtifactsBinPath)\GitExtSshAskPass\GitExtSshAskPass.exe" />
	</Component>

	Condition="!$(Arm64Build)"
	Condition=" '$(Arm64Build)' != 'true' "

Add support to build GE for Windows on Arm64 (WoA) #10649

Add support to build GE for Windows on Arm64 (WoA) #10649

Conversation

chirontt commented Jan 15, 2023 • edited by RussKie

Proposed changes

Test methodology

Test environment(s)

Merge strategy

RussKie commented Jan 16, 2023

RussKie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RussKie commented Jan 17, 2023 via email • edited by ghost

chirontt commented Jan 23, 2023

RussKie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chirontt Mar 13, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chirontt Mar 13, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RussKie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RussKie left a comment

Choose a reason for hiding this comment

RussKie commented Apr 21, 2023

ghost commented May 5, 2023

AArnott commented May 17, 2023

RussKie commented May 18, 2023 via email • edited by ghost

AArnott commented May 18, 2023

RussKie commented May 18, 2023 via email • edited by ghost

AArnott commented May 18, 2023

vbjay commented May 18, 2023

RussKie commented May 18, 2023 via email • edited by ghost

RussKie commented May 18, 2023 via email • edited by ghost

AArnott commented May 18, 2023

vbjay commented May 18, 2023

AArnott commented May 19, 2023

RussKie commented May 21, 2023

AArnott commented May 21, 2023

RussKie commented May 22, 2023

chirontt commented Jan 15, 2023 •

edited by RussKie

RussKie commented Jan 17, 2023 via email •

edited by ghost

chirontt Mar 13, 2023 •

edited

chirontt Mar 13, 2023 •

edited

RussKie commented May 18, 2023 via email •

edited by ghost

RussKie commented May 18, 2023 via email •

edited by ghost

RussKie commented May 18, 2023 via email •

edited by ghost

RussKie commented May 18, 2023 via email •

edited by ghost