kvserver: 23.2 replication microbenchmark regressions #111561

erikgrinaker · 2023-10-02T08:35:47Z

Preliminary results were posted in https://docs.google.com/spreadsheets/d/1xYiEZlqL9tukSgZENvZIxgIHxrwo7c_PKFDdXCP6QkA/edit#gid=5. These may be fairly old, but we should look into the replication-related regressions until newer results are ready. These may or may not be false positives. In no particular order:

kvserver/ReplicaProposal @pavelkalinnikov
- clusterversion: benchmark and rm Unmarshal allocs #113043
- kvserver: do lazy map allocations in replicaFlowControl #113150
- kv,server,roachpb: avoid error overhead for x-locality comparison #113229
- logstore: pool MVCCStats to avoid hot path allocs #113742
- Performance diff is within a reasonable error margin.
kvserver/BumpSideTransportClosed @erikgrinaker
- 15% regression, unknown cause, small in absolute terms
- Benchmark sensitive to Go runtime: regressed from 220ns on 1.19 to 250ns on 1.20; went back to 220ns on 1.21.
kvserver/raftentry/EntryCache @pavelkalinnikov
- single-digit % CPU regression. Hard to attribute to anything except Go runtime changes
- Quite a bit more time spent in raftpb.(*Entry).Size(): 3.62s -> 4.48s
kvserver/raftentry/EntryCacheClearTo @pavelkalinnikov
- Same as kvserver/raftentry/EntryCache
- Particularly, more time is spent in proto size computation
kvserver/logstore/LogStore_StoreEntries @pavelkalinnikov
- clusterversion: benchmark and rm Unmarshal allocs #113043
- One more finding, fixed in logstore: pool MVCCStats to avoid hot path allocs #113742.
kvserver/StoreRangeMerge @erikgrinaker
- ~10% allocs/op increase, but highly unreliable -- alloc/op and time decreased, and we don't particularly care about allocs/op here anyway.
- @pavelkalinnikov measured after all the fixes. Allocations dropped to roughly level; but the 23.1 measurement is unreliable. CPU is still up by a couple of percent, but that's probably fine.
kvserver/closedts/tracker/HeapTracker @erikgrinaker
- fixed in recent benchmarks

Jira issue: CRDB-31969

The text was updated successfully, but these errors were encountered:

blathers-crl · 2023-10-02T08:35:50Z

cc @cockroachdb/replication

erikgrinaker · 2023-10-03T15:22:29Z

Weekly microbenchmark results are available in #perf-ops.

erikgrinaker · 2023-10-04T08:32:55Z

Here's the latest KV microbenchmark results (compared with 23.1): https://docs.google.com/spreadsheets/d/12cla5YkEpAd1gX3nMushoN354yuurO67k7PfPs5wI64/edit#gid=5

blathers-crl · 2023-10-23T12:14:04Z

Hi @erikgrinaker, please add branch-* labels to identify which branch(es) this release-blocker affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

erikgrinaker · 2023-10-24T20:54:20Z

Looked at BumpSideTransportClosed. No significant profile changes nor code changes -- Go runtime change? ~15% diff, small in absolute terms, but higher variance.

name                        old time/op  new time/op  delta
BumpSideTransportClosed-24   225ns ± 0%   261ns ±11%  +16.38%  (p=0.000 n=9+10)

Before: cpu-before.pprof.gz

After: cpu-after.pprof.gz

pav-kv · 2023-10-25T12:17:06Z

For kvserver/logstore/LogStore_StoreEntries allocations regression, there is a difference between 23.1 and master:

We're seeing that allocations doubled to 2 allocs/op because there is a new allocation in clusterversion package. Previously, the logstore.newStoreEntriesBatch call did not allocate.

@jbowens Has storage.newPebbleBatch started using this version check recently?

Something here allocates:

cockroach/pkg/clusterversion/setting.go

Lines 132 to 144 in a0b6f5c

    
           func (cv *clusterVersionSetting) activeVersionOrEmpty( 
        
           	ctx context.Context, sv *settings.Values, 
        
           ) ClusterVersion { 
        
           	encoded := cv.GetInternal(sv) 
        
           	if encoded == nil { 
        
           		return ClusterVersion{} 
        
           	} 
        
           	var curVer ClusterVersion 
        
           	if err := protoutil.Unmarshal(encoded.([]byte), &curVer); err != nil { 
        
           		log.Fatalf(ctx, "%v", err) 
        
           	} 
        
           	return curVer 
        
           }

This potentially impacts other benchmarks too, as creating a Pebble batch is a common operation.

More generally, looks like every version check in CRDB allocates? Opened a PR #113043 to this extent.

jbowens · 2023-10-25T14:04:31Z

@jbowens Has storage.newPebbleBatch started using this version check recently?

Yeah, that version check is new in 23.2. It's unfortunate that allocates. I suppose we could subscribe to cluster version progressions (IIRC there's an API to do this), and stash the current cluster version on the Engine for use in newBatch.

pav-kv · 2023-10-25T14:05:23Z

@jbowens I think #113043 should fix the thing.

erikgrinaker · 2023-10-26T11:36:01Z

I wonder if it's worth running some of these benchmarks with different Go versions, since we're not able to find clear explanations for a couple of these regressions. If there is a performance regression in Go itself then we'd probably want to know about that.

pav-kv · 2023-10-26T11:49:25Z

I wonder if it's worth running some of these benchmarks with different Go versions, since we're not able to find clear explanations for a couple of these regressions. If there is a performance regression in Go itself then we'd probably want to know about that.

Also could be worth checking whether there is a change in proto. The raft cache tests spend a bit more time in protos.

pav-kv · 2023-10-26T12:25:29Z

In ReplicaProposal benchmark (which is more like an end-to-end thing), the IsActive allocations add up as 1% of all allocations:

The remaining ~10% difference is yet to be explained.

Raft stack almost doubled the allocations:

Before/after

The notable addition is updateProposalQuotaRaftMuLocked. Added recently with the replication admission control? CC @sumeerbhola

Another one is the error handling bonanza in HandleRaftRequestSent. Added recently in #105122 and #105267. Gut feeling is that we should not see these errors in this test. CC @kvoli as reviewer of these PRs - if you remember the context, are these errors in the profile legit?

pav-kv · 2023-10-26T13:30:10Z

@erikgrinaker Given the above, I suspect that Raft stack cost got increased, so the thing you observed in kvserver/StoreRangeMerge is probably impacted too.

sumeerbhola · 2023-10-26T14:27:20Z

The notable addition is updateProposalQuotaRaftMuLocked. Added recently with the replication admission control? CC @sumeerbhola

Can you post a link to the "after" memory profile file?

aadityasondhi · 2023-10-26T14:53:46Z

The notable addition is updateProposalQuotaRaftMuLocked. Added recently with the replication admission control?

Took a quick look in the code. Tracking those down updateProposalQuotaRaftMuLocked, I see most of the new allocs come from this bit of code with all the get functions. Maybe we can do something about it.

https://github.com/cockroachdb/cockroach/blob/master/pkg/kv/kvserver/flow_control_replica_integration.go#L298-L302

These methods will typically return empty maps, so we can avoid the allocations. Informs cockroachdb#111561 Epic: none Release note: None

kvoli · 2023-10-26T15:49:20Z

Another one is the error handling bonanza in HandleRaftRequestSent. Added recently in #105122 and #105267. Gut feeling is that we should not see these errors in this test. CC @kvoli as reviewer of these PRs - if you remember the context, are these errors in the profile legit?

Unlikely to see the errors in actual deployments as they are created for missing/unexpected locality specification.

We can drop them, seeing they are used only in VInfof(5,...) -- and either check V(5) before doing work, or entirely.

113150: kvserver: do lazy map allocations in replicaFlowControl r=pavelkalinnikov,aadityasondhi a=sumeerbhola These methods will typically return empty maps, so we can avoid the allocations. Informs #111561 Epic: none Release note: None Co-authored-by: sumeerbhola <sumeer@cockroachlabs.com>

These methods will typically return empty maps, so we can avoid the allocations. Informs #111561 Epic: none Release note: None

Cross locality traffic instrumentation was added to raft, snapshots and batch requests to quantify the amount of cross region/zone traffic. Errors would be returned from `CompareWithLocality` when the region, or zone locality flags were set in an unsupported manner according to our documentation. These error allocations added overhead (cpu/mem) when hit. Alter `CompareWithLocality` to return booleans in place of an error to reduce overhead. Resolves: cockroachdb#111148 Resolves: cockroachdb#111142 Informs: cockroachdb#111561 Release note: None

113069: kvserver: add BenchmarkNodeLivenessScanStorage to measure liveness scan r=andrewbaptist,jbowens a=sumeerbhola Node liveness scans, like the one done in MaybeGossipNodeLivenessRaftMuLocked, while holding raftMu, are performance sensitive, and slowness has caused production issues (https://github.com/cockroachlabs/support/issues/2665, https://github.com/cockroachlabs/support/issues/2107). This benchmark measures the scan performance both when DELs (due to GC) have not been compacted away, and when they have. It also sets up a varying number of live versions since decommissioned nodes will have a single live version. Results on M1 macbook on master with dead-keys=false and compacted=true: ``` NodeLivenessScanStorage/num-live=2/compacted=true-10 26.80µ ± 9% NodeLivenessScanStorage/num-live=5/compacted=true-10 30.34µ ± 3% NodeLivenessScanStorage/num-live=10/compacted=true-10 38.88µ ± 8% NodeLivenessScanStorage/num-live=1000/compacted=true-10 861.5µ ± 3% ``` When compacted=false the scan takes ~10ms, which is > 100x slower, but probably acceptable for this workload. ``` NodeLivenessScanStorage/num-live=2/compacted=false-10 9.430m ± 5% NodeLivenessScanStorage/num-live=5/compacted=false-10 9.534m ± 4% NodeLivenessScanStorage/num-live=10/compacted=false-10 9.456m ± 2% NodeLivenessScanStorage/num-live=1000/compacted=false-10 10.34m ± 7% ``` dead-keys=true (and compacted=false) defeats the NextPrefix optimization, since the next prefix can have all its keys deleted and the iterator has to step through all of them (it can't be sure that all the keys for that next prefix are deleted). This case should not occur in the liveness range, as we don't remove decommissioned entries, but is included for better understanding. ``` NodeLivenessScanStorage/num-live=2/dead-keys=true/compacted=false-10 58.33m ``` Compared to v22.2, the results are sometimes > 10x faster, when the pebbleMVCCScanner seek optimization in v22.2 was defeated. ``` │ sec/op │ sec/op vs base │ NodeLivenessScanStorage/num-live=2/compacted=false-10 117.280m ± 2% 9.430m ± 5% -91.96% (p=0.002 n=6) NodeLivenessScanStorage/num-live=5/compacted=false-10 117.298m ± 0% 9.534m ± 4% -91.87% (p=0.002 n=6) NodeLivenessScanStorage/num-live=10/compacted=false-10 12.009m ± 0% 9.456m ± 2% -21.26% (p=0.002 n=6) NodeLivenessScanStorage/num-live=1000/compacted=false-10 13.04m ± 0% 10.34m ± 7% -20.66% (p=0.002 n=6) │ block-bytes/op │ block-bytes/op vs base │ NodeLivenessScanStorage/num-live=2/compacted=false-10 14.565Mi ± 0% 8.356Mi ± 0% -42.63% (p=0.002 n=6) NodeLivenessScanStorage/num-live=5/compacted=false-10 14.570Mi ± 0% 8.361Mi ± 0% -42.61% (p=0.002 n=6) NodeLivenessScanStorage/num-live=10/compacted=false-10 11.094Mi ± 0% 8.368Mi ± 0% -24.57% (p=0.002 n=6) NodeLivenessScanStorage/num-live=1000/compacted=false-10 12.235Mi ± 0% 8.990Mi ± 0% -26.53% (p=0.002 n=6) │ B/op │ B/op vs base │ NodeLivenessScanStorage/num-live=2/compacted=false-10 42.83Ki ± 4% 41.87Ki ± 0% -2.22% (p=0.002 n=6) NodeLivenessScanStorage/num-live=5/compacted=false-10 43.28Ki ± 3% 41.84Ki ± 0% -3.32% (p=0.002 n=6) NodeLivenessScanStorage/num-live=10/compacted=false-10 37.59Ki ± 0% 41.92Ki ± 0% +11.51% (p=0.002 n=6) NodeLivenessScanStorage/num-live=1000/compacted=false-10 37.67Ki ± 1% 42.66Ki ± 0% +13.23% (p=0.002 n=6) │ allocs/op │ allocs/op vs base │ NodeLivenessScanStorage/num-live=2/compacted=false-10 105.00 ± 8% 85.00 ± 0% -19.05% (p=0.002 n=6) NodeLivenessScanStorage/num-live=5/compacted=false-10 107.00 ± 5% 85.00 ± 0% -20.56% (p=0.002 n=6) NodeLivenessScanStorage/num-live=10/compacted=false-10 74.00 ± 1% 85.00 ± 0% +14.86% (p=0.002 n=6) NodeLivenessScanStorage/num-live=1000/compacted=false-10 79.00 ± 1% 92.00 ± 1% +16.46% (p=0.002 n=6) ``` Relates to https://github.com/cockroachlabs/support/issues/2665 Epic: none Release note: None 113229: kv,server,roachpb: avoid error overhead for x-locality comparison r=pavelkalinnikov a=kvoli Cross locality traffic instrumentation was added to raft, snapshots and batch requests to quantify the amount of cross region/zone traffic. Errors would be returned from `CompareWithLocality` when the region, or zone locality flags were set in an unsupported manner according to our documentation. These error allocations added overhead (cpu/mem) when hit. Alter `CompareWithLocality` to return booleans in place of an error to reduce overhead. Resolves: #111148 Resolves: #111142 Informs: #111561 Release note: None Co-authored-by: sumeerbhola <sumeer@cockroachlabs.com> Co-authored-by: Austen McClernon <austen@cockroachlabs.com>

Cross locality traffic instrumentation was added to raft, snapshots and batch requests to quantify the amount of cross region/zone traffic. Errors would be returned from `CompareWithLocality` when the region, or zone locality flags were set in an unsupported manner according to our documentation. These error allocations added overhead (cpu/mem) when hit. Alter `CompareWithLocality` to return booleans in place of an error to reduce overhead. Resolves: #111148 Resolves: #111142 Informs: #111561 Release note: None

erikgrinaker · 2023-10-31T08:34:46Z

@pavelkalinnikov We're planning to cut the .0 branch today, can we get the remaining fixes backported before then?

pav-kv · 2023-10-31T08:42:07Z

There is only #113043. I probably can workaround the linter and land it today.

113043: clusterversion: benchmark and rm Unmarshal allocs r=erikgrinaker a=pavelkalinnikov Currently, each `IsActive` does a memory allocation: ``` ==================== Test output for //pkg/clusterversion:clusterversion_test: goos: darwin goarch: arm64 BenchmarkClusterVersionSettingIsActive BenchmarkClusterVersionSettingIsActive-10 28778041 42.03 ns/op 16 B/op 1 allocs/op PASS ``` Since the cluster version check is in many hot paths, we should eliminate this allocation. After: ``` ==================== Test output for //pkg/clusterversion:clusterversion_test: goos: darwin goarch: arm64 BenchmarkClusterVersionSettingIsActive BenchmarkClusterVersionSettingIsActive-10 45417914 26.43 ns/op 0 B/op 0 allocs/op PASS ``` Touches #111561 Epic: none Release note: none Co-authored-by: Pavel Kalinnikov <pavel@cockroachlabs.com>

pav-kv · 2023-10-31T19:10:51Z

New results from this Monday. These don't yet contain the proto Unmarshal fix, but some/all other optimizations should be in already.

There is still some room for improvement.

pav-kv · 2023-11-02T12:17:53Z

Taking another (first one was here) look at LogStore_StoreEntries after all the fixes.

23.1:

BenchmarkLogStore_StoreEntries/bytes=1.0_KiB
BenchmarkLogStore_StoreEntries/bytes=1.0_KiB-24                   924716              1267 ns/op             157 B/op          0 allocs/op
BenchmarkLogStore_StoreEntries/bytes=256_KiB
BenchmarkLogStore_StoreEntries/bytes=256_KiB-24                    31063             38715 ns/op             239 B/op          0 allocs/op
BenchmarkLogStore_StoreEntries/bytes=512_KiB
BenchmarkLogStore_StoreEntries/bytes=512_KiB-24                     9567            121584 ns/op             558 B/op          0 allocs/op
BenchmarkLogStore_StoreEntries/bytes=1.0_MiB
BenchmarkLogStore_StoreEntries/bytes=1.0_MiB-24                     1256            855636 ns/op         2207006 B/op          4 allocs/op
BenchmarkLogStore_StoreEntries/bytes=2.0_MiB
BenchmarkLogStore_StoreEntries/bytes=2.0_MiB-24                      572           1848153 ns/op         4655889 B/op          5 allocs/op

master:

BenchmarkLogStore_StoreEntries/bytes=1.0_KiB
BenchmarkLogStore_StoreEntries/bytes=1.0_KiB-24                   705525              1429 ns/op             333 B/op          1 allocs/op
BenchmarkLogStore_StoreEntries/bytes=256_KiB
BenchmarkLogStore_StoreEntries/bytes=256_KiB-24                    28836             40695 ns/op             402 B/op          1 allocs/op
BenchmarkLogStore_StoreEntries/bytes=512_KiB
BenchmarkLogStore_StoreEntries/bytes=512_KiB-24                     9249            120265 ns/op             576 B/op          1 allocs/op
BenchmarkLogStore_StoreEntries/bytes=1.0_MiB
BenchmarkLogStore_StoreEntries/bytes=1.0_MiB-24                     1346            844536 ns/op         2187245 B/op          4 allocs/op
BenchmarkLogStore_StoreEntries/bytes=2.0_MiB
BenchmarkLogStore_StoreEntries/bytes=2.0_MiB-24                      645           1716122 ns/op         4522320 B/op          5 allocs/op

The "big" tests are even. The "small" tests allocate a bit more, and therefore are slower. Figuring out the reason now.

The clear outlier is the logstore.logAppend:

Before/after mem profiles

The reason (I double checked before/after) is #107680. Probably it added added an allocation here (size of MVCCStats is ~168 bytes which matches the diff):

cockroach/pkg/kv/kvserver/logstore/logstore.go

Lines 386 to 388 in e836704

    
           opts := storage.MVCCWriteOptions{ 
        
           	Stats: &diff, 
        
           }

@erikgrinaker do you think there is a quick fix? One way would be returning the stats delta rather than providing an "output" parameter. But that's not quick, and might have cons.

erikgrinaker · 2023-11-03T09:18:15Z

I'm not sure we necessarily care about this allocation. The IO cost of log appends is probably high enough that the allocation doesn't make a significant difference, and this cost is constant for all appended entries. This benchmark uses an in-memory engine, which hides the IO cost (and we're only benchmarking the inner logic).

I suppose we could pool these to amortize the allocation cost though.

pav-kv · 2023-11-03T09:28:43Z

Yeah, feel similarly. I guess we can leave it then, just need to be mindful that it might impact other microbenchmarks that are yet to be investigated.

erikgrinaker · 2023-11-03T09:36:57Z

Can you try to pool them and see if it makes any difference on BenchmarkKV/Insert in pkg/sql/tests?

113742: logstore: pool MVCCStats to avoid hot path allocs r=erikgrinaker a=pavelkalinnikov Before this commit, `logstore.logAppend` allocated `MVCCStats` on heap despite its lifetime being confined within this function's scope. This commit switches to allocating `MVCCStats` on a `sync.Pool`. Since `logAppend` already has another pooled `roachpb.Value` with exactly the same lifetime, coalesce the allocation of both and put them in the same pool. This makes the cost of this change zero. Microbenchmark results: ``` // before BenchmarkLogStore_StoreEntries/bytes=1.0_KiB-24 837254 1270 ns/op 333 B/op 1 allocs/op // after BenchmarkLogStore_StoreEntries/bytes=1.0_KiB-24 1000000 1153 ns/op 157 B/op 0 allocs/op ``` Part of #111561 Epic: none Release note: none Co-authored-by: Pavel Kalinnikov <pavel@cockroachlabs.com>

pav-kv · 2023-11-03T17:02:06Z

Turns out there is a simple fix, so we're just fixing the thing.
#113742

pav-kv · 2023-11-07T09:11:36Z

The repl-specific part is done. If there is any Go-/proto-induced regression, the impact of it should be investigated on a broader scope.

erikgrinaker added C-performance Perf of queries or internals. Solution not expected to change functional behavior. A-kv-replication Relating to Raft, consensus, and coordination. T-kv-replication KV Replication Team labels Oct 2, 2023

erikgrinaker added branch-master Failures on the master branch. GA-blocker labels Oct 2, 2023

exalate-issue-sync bot assigned aliher1911 Oct 3, 2023

exalate-issue-sync bot removed branch-master Failures on the master branch. GA-blocker labels Oct 3, 2023

aliher1911 removed their assignment Oct 11, 2023

exalate-issue-sync bot assigned pav-kv Oct 13, 2023

erikgrinaker added the GA-blocker label Oct 23, 2023

erikgrinaker added the branch-release-23.2 Used to mark GA and release blockers and technical advisories for 23.2 label Oct 23, 2023

pav-kv mentioned this issue Oct 25, 2023

clusterversion: benchmark and rm Unmarshal allocs #113043

Merged

pav-kv mentioned this issue Oct 26, 2023

server: regression in BenchmarkColBatchScan due to x-region, x-zone metrics #111148

Closed

sumeerbhola added a commit to sumeerbhola/cockroach that referenced this issue Oct 26, 2023

kvserver: do lazy map allocations in replicaFlowControl

56b5c2f

These methods will typically return empty maps, so we can avoid the allocations. Informs cockroachdb#111561 Epic: none Release note: None

sumeerbhola mentioned this issue Oct 26, 2023

kvserver: do lazy map allocations in replicaFlowControl #113150

Merged

blathers-crl bot mentioned this issue Oct 27, 2023

release-23.2: kvserver: do lazy map allocations in replicaFlowControl #113217

Merged

blathers-crl bot pushed a commit that referenced this issue Oct 27, 2023

kvserver: do lazy map allocations in replicaFlowControl

78bb776

These methods will typically return empty maps, so we can avoid the allocations. Informs #111561 Epic: none Release note: None

kvoli mentioned this issue Oct 27, 2023

kv,server,roachpb: avoid error overhead for x-locality comparison #113229

Merged

blathers-crl bot mentioned this issue Oct 30, 2023

release-23.2: kv,server,roachpb: avoid error overhead for x-locality comparison #113302

Merged

pav-kv mentioned this issue Oct 31, 2023

clusterversion: proto Unmarshal on each version check #113385

Closed

This was referenced Oct 31, 2023

release-23.1: clusterversion: benchmark and rm Unmarshal allocs #113392

Closed

release-23.2: clusterversion: benchmark and rm Unmarshal allocs #113393

Merged

pav-kv mentioned this issue Nov 3, 2023

logstore: pool MVCCStats to avoid hot path allocs #113742

Merged

blathers-crl bot mentioned this issue Nov 3, 2023

release-23.2: logstore: pool MVCCStats to avoid hot path allocs #113760

Merged

pav-kv closed this as completed Nov 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kvserver: 23.2 replication microbenchmark regressions #111561

kvserver: 23.2 replication microbenchmark regressions #111561

erikgrinaker commented Oct 2, 2023 •

edited by pav-kv

blathers-crl bot commented Oct 2, 2023

erikgrinaker commented Oct 3, 2023

erikgrinaker commented Oct 4, 2023

blathers-crl bot commented Oct 23, 2023

erikgrinaker commented Oct 24, 2023

pav-kv commented Oct 25, 2023 •

edited

jbowens commented Oct 25, 2023

pav-kv commented Oct 25, 2023

erikgrinaker commented Oct 26, 2023

pav-kv commented Oct 26, 2023

pav-kv commented Oct 26, 2023 •

edited

pav-kv commented Oct 26, 2023 •

edited

sumeerbhola commented Oct 26, 2023

aadityasondhi commented Oct 26, 2023

kvoli commented Oct 26, 2023

erikgrinaker commented Oct 31, 2023

pav-kv commented Oct 31, 2023

pav-kv commented Oct 31, 2023 •

edited

pav-kv commented Nov 2, 2023 •

edited

erikgrinaker commented Nov 3, 2023

pav-kv commented Nov 3, 2023

erikgrinaker commented Nov 3, 2023

pav-kv commented Nov 3, 2023

pav-kv commented Nov 7, 2023

kvserver: 23.2 replication microbenchmark regressions #111561

kvserver: 23.2 replication microbenchmark regressions #111561

Comments

erikgrinaker commented Oct 2, 2023 • edited by pav-kv

blathers-crl bot commented Oct 2, 2023

erikgrinaker commented Oct 3, 2023

erikgrinaker commented Oct 4, 2023

blathers-crl bot commented Oct 23, 2023

erikgrinaker commented Oct 24, 2023

pav-kv commented Oct 25, 2023 • edited

jbowens commented Oct 25, 2023

pav-kv commented Oct 25, 2023

erikgrinaker commented Oct 26, 2023

pav-kv commented Oct 26, 2023

pav-kv commented Oct 26, 2023 • edited

pav-kv commented Oct 26, 2023 • edited

sumeerbhola commented Oct 26, 2023

aadityasondhi commented Oct 26, 2023

kvoli commented Oct 26, 2023

erikgrinaker commented Oct 31, 2023

pav-kv commented Oct 31, 2023

pav-kv commented Oct 31, 2023 • edited

pav-kv commented Nov 2, 2023 • edited

erikgrinaker commented Nov 3, 2023

pav-kv commented Nov 3, 2023

erikgrinaker commented Nov 3, 2023

pav-kv commented Nov 3, 2023

pav-kv commented Nov 7, 2023

erikgrinaker commented Oct 2, 2023 •

edited by pav-kv

pav-kv commented Oct 25, 2023 •

edited

pav-kv commented Oct 26, 2023 •

edited

pav-kv commented Oct 26, 2023 •

edited

pav-kv commented Oct 31, 2023 •

edited

pav-kv commented Nov 2, 2023 •

edited