Add `util/groupcommit` package in `core` subproject #1607

komamitsu · 2024-03-14T10:53:19Z

Description

ScalarDB may sometimes need to work with a high latency storage (e.g., multi-region replicated database). In such a case, the throughput of application using ScalarDB would be affected by the high latency transactions. With a group commit mechanism, high latency issue can be hidden with concurrent transactions.

In this PR, we introduce util/groupcommit package in core subproject which provides a mechanism to support the ScalarDB protocol. The package alone doesn't affect the existing implementations. A following PR will integrate the new package and the existing implementations.

Related issues and/or PRs

N/A

Changes made

This PR basically adds the following classes.

The util/groupcommit package has a concept about keys and they should be described before looking into the classes.

Keys
- Child Key : This is passed by a client (e.g., UUID v4 like ScalarDB transaction ID)
- Parent Key : This is generated by KeyManipulator class which is injected by a client and contains logics to handle Keys
- Full Key : This consists of Child Key and Parent Key

The main components introduced by this PR are as follows:

Group
- A unit of (group) commit
- Contains slots (see below about slots). The number of slots in a group is decided by the configuration
- Possible status are as follows:
  - OPEN : Initial status. Accepting new slot reservation as the number of slots isn't fixed yet
  - CLOSED : Not accepting new slot reservations since the number of slots is already fixed. Waiting all the slots are set with values
  - READY : All the slots are set with values. Ready to commit
  - DONE : Group commit is done and all the clients have get the results
- There are 2 types of Groups
  - NormalGroup
    - Contains multiple slots and commits them at once. Ideally only this Group should be used in terms of performance
    - Parent Key is used to identify NormalGroup
  - DelayedGroup
    - Contains only a single slot which is delayed and removed from the NormalGroup
    - Full Key is used to identify DelayedGroup
Slot
- A container for a value. Values are supposed to be emitted at once by this group commit mechanism
- Each Slot has the corresponding Child Key
GroupManager
- Manages all the ongoing Groups
- It internally contains 2 maps
  - NormalGroupMap : Key=Parent-Key, Value=NormalGroup
  - DelayedGroupMap : Key=Full-Key, Value=DelayedGroup
Worker threads
- Group-Close Thread : Fixes the number of slot (or close the group) when the Group is timed out
- Delayed-Slot-Move Thread : Moves delayed Slots from NormalGroupMap to DelayedGroupMap
- Group-Cleanup Thread : Removes Group entries that are already done from NormalGroupMap or DelayedGroupMap. It's a sort of garbage collector
GroupCommitter
- Manages all the above classes
- Has the following public APIs
  - reserve(Child Key) returns Full Key
    - Allocates a Slot in the current group that is associated with a Full Key
    - The allocated slot status is initially NOT_READY
  - ready(Full Key)
    - Marks the slot associated with the Full Key READY
    - Waits until all the slots in the group are READY
    - Internally calls the Emittable.emit() which is injected by the client with either of the following keys once the group gets READY
      - Parent Key : In most cases, this type of Key is used to emit multiple values in NormalGroup
      - Full Key : This type of Key is used only when it takes long time to call this API and the delayed slot is separated from the original NormalGroup to a new DelayedGroup to prevent other slots in the Group from being delayed

Very rough overall architecture of the group commit

Replace txid with Key when reading the description.

Lifecycle of Group

Checklist

The following is a best-effort checklist. If any items in this checklist are not applicable to this PR or are dependent on other, unmerged PRs, please still mark the checkboxes after you have read and understood each item.

I have commented my code, particularly in hard-to-understand areas.
I have updated the documentation to reflect the changes.
Any remaining open issues linked to this PR are documented and up-to-date (Jira, GitHub, etc.).
Tests (unit, integration, etc.) have been added for the changes.
My changes generate no new warnings.
Any dependent changes in other PRs have been merged and published.

Additional notes (optional)

The current implementation contains the following 3 TODOs. They'll be handled in other PRs.

Possible performance improvement with tweaking the scope of locks
Replace the tentative metrics monitoring with DropWizard's one
Performance improvement of DelayedSlotMoveWorker
- The worker basically waits for every item, so the performance improvement is required
- Introducing a priority queue is an option, which was suggested by @brfrn169

Release notes

N/A (This PR itself isn't a user facing change).

brfrn169

My first review is complete, and I've left several comments. Please take a look when you have time! I will conduct a second review later on. Thank you for your great work!

brfrn169 · 2024-03-21T14:41:43Z

core/src/main/java/com/scalar/db/util/groupcommit/GroupCommitter.java

+  // TODO: This should be replaced by other metrics mechanism.
+  private void startMonitorExecutorService() {
+    Runnable print =
+        () -> logger.info("[MONITOR] Timestamp={}, Metrics={}", Instant.now(), getMetrics());


I feel we can make it a debug log. What do you think?

Sounds good! I'll update it.

BTW, this will be soon replaced with #1614 which uses DropWizard Metrics.

Fixed in ade2864

brfrn169 · 2024-03-22T06:05:43Z

core/src/main/java/com/scalar/db/util/groupcommit/Group.java

+  private void reserveSlot(Slot<PARENT_KEY, CHILD_KEY, FULL_KEY, EMIT_KEY, V> slot) {
+    Slot<?, ?, ?, ?, ?> oldSlot = slots.put(slot.key(), slot);
+    if (oldSlot != null) {
+      logger.warn("An old slot exist unexpectedly. {}", oldSlot.fullKey());


This shouldn't occur, right? If so, just showing the warning message is okay?

Good catch! Right. I'll update it to throw AssertionError or IllegalStateException.

Fixed in 8d96fd4

brfrn169 · 2024-03-22T06:14:51Z

core/src/main/java/com/scalar/db/util/groupcommit/GroupCommitter.java

+   *
+   * @param fullKey A full key to specify the slot.
+   */
+  public void remove(FULL_KEY fullKey) {


When is this remove method intended to be called?

Good question.

Let's say a transaction executes some CRUD operations, starts PREPARE, and fails before the group commit for coordinator state. In this case, a group commit slot is allocated for the transaction and needed to be removed by calling this method not to leave garbage slots.

Updated the comment in 4ff928a

brfrn169 · 2024-03-22T06:22:02Z

core/src/main/java/com/scalar/db/util/groupcommit/GroupCommitter.java

+   * @param value A value to be set to the slot.
+   * @throws GroupCommitException when group commit fails
+   */
+  public void ready(FULL_KEY fullKey, V value) throws GroupCommitException {


What is the purpose of this value argument? Maybe we use it for a transaction ID (a child key) for each transaction within a group?

In the case of coordinator state commit, V is supposed to be Snapshot and a passed value will be committed with other snapshots in the same group.

Updated the comment to make it clearer in fa5e726

brfrn169 · 2024-03-22T06:33:19Z

core/src/main/java/com/scalar/db/util/groupcommit/GroupCommitter.java

+ * @param <EMIT_KEY> A key type that Emitter can interpret.
+ * @param <V> A value type to be set to a slot.
+ */
+public class GroupCommitter<PARENT_KEY, CHILD_KEY, FULL_KEY, EMIT_KEY, V> implements Closeable {


I think we can put @ThreadSafe or @NotThreadSafe for each class.

Fixed in e581918

and simplify some code

Fix a race condition bug that a newly created DelayedGroup is already closed when reserving a slot. The scenario that the issue occurs is as follows - DelayedSlotMoveWorker adds the new DelayedGroup into GroupCleanupWorker - GroupCleanupWorker calls DelayedGroup.updateState() before DelayedSlotMoveWorker reserves a slot on the DelayedGroup - The slots of the DelayedGroup is still empty and it’s marked as DONE - DelayedSlotMoveWorker fails to reserve a slot on the DelayedGroup since it’s closed (DONE)

feeblefakie

Thank you for the great improvement and very sorry for the late review.
I took a brief look and left some comments and questions. PTAL!
(Let me check this one more time.)

Let me ask additional questions here.

We discussed before that we might need to access the coordinator table twice in some recovery cases, but how does the current code handle the case?
If one transaction in a group fails to commit, will the transaction be handled in a DelayedGroup?

feeblefakie · 2024-03-27T10:21:37Z

core/src/main/java/com/scalar/db/util/groupcommit/Group.java

+    // Waiting all the slots are set with values.
+    //
+    // Groups with OPEN status can move to CLOSED.
+    CLOSED(true, false, false),


CLOSED corresponds to OPEN so it's good, but it sounds like the group has already finished processing.
How about FIXED?

Thanks for the suggestion! Actually it was first like SIZE-FIXED and I changed it to CLOSED to sound to reject new requests. But as you said, it also sounds things are done. I'll update it to use FIX.

Fixed in 5000335. Used SIZE_FIXED to make it clearer.

feeblefakie · 2024-03-27T10:38:01Z

core/src/main/java/com/scalar/db/util/groupcommit/GroupManager.java

+  // If it returns null, the Group is already closed and a retry is needed.
+  @Nullable
+  FULL_KEY reserveNewSlot(CHILD_KEY childKey) {
+    long stamp = lock.writeLock();


I might miss something, but since we have only one active group at a time, I'm wondering how much it badly affects performance if there are many concurrent transactions.
We could use multiple active groups to mitigate the potential issue, but it might complicate the mechanism.
Any thoughts on this?

Good point. I also thought and tried multiple current (== active) groups before. But, the performance with multiple active groups didn't have a good performance. I think how fast to fill up a group with slots (and values) is a very important factor to move it to the next state (i.e., ready to commit) in the group commit and using multiple active groups doesn't fit.

I benchmarked this util/groupcommit pacakge itself and the average throughput was about 65K TPS without any wait while the throughput with some delays to emulate actual latencies of underlying database had far low throughputs. It means current single active group won't be a bottleneck in usual usages of ScalarDB, I think.

Thank you for the detailed explanation!
It makes sense. Let's leave it as is.

komamitsu · 2024-03-28T02:39:33Z

@feeblefakie Thanks for the questions!

We discussed before that we might need to access the coordinator table twice in some recovery cases, but how does the current code handle the case?

This PR contains doesn't contain any integration with the coordinator state commit, so let me refer to another PR's change (https://github.com/scalar-labs/scalardb/pull/1523/files#diff-8049e66d11b81cdd5a1dc82c4977b5336741bcee24dd7ddd54c6a6940f09497bR83-R89). I tried the idea to put child-tx-id in a clustering key before in order to make it possible to fetch both group-committed-transactions' ID and separately-committed-transactions' ID. But, there wasn't significant performance improvement with YCSB-F benchmark despite it would require cumbersome table schema migrations. So, current implementation is to access the coordinator table twice.

If one transaction in a group fails to commit, will the transaction be handled in a DelayedGroup?

No. Slots are moved to DelayedGroup only when it's delayed. If a NormalGroup fails to commit due to network or underlying database issues, all the slots in the group fail. BTW, it's about Commit-State phase. If a transaction fails in Prepare or Validate, it'll be just removed from the NormalGroup without being moved to DelayedGroup.

in GroupCommitterConcurrentTest to see if the timeout mechanisms work

feeblefakie · 2024-04-01T23:37:26Z

@komamitsu

I tried the idea to put child-tx-id in a clustering key before in order to make it possible to fetch both group-committed-transactions' ID and separately-committed-transactions' ID. But, there wasn't significant performance improvement with YCSB-F benchmark despite it would require cumbersome table schema migrations. So, current implementation is to access the coordinator table twice.

Thank you for the explanation.
I think how much performance improves depends on how much the workload triggers conflicts,
so it might be worth trying TPC-C to see how it goes since there are not many conflicts in YCSB.

No. Slots are moved to DelayedGroup only when it's delayed. If a NormalGroup fails to commit due to network or underlying database issues, all the slots in the group fail. BTW, it's about Commit-State phase. If a transaction fails in Prepare or Validate, it'll be just removed from the NormalGroup without being moved to DelayedGroup.

Thank you!
I got it.

brfrn169 · 2024-04-02T06:59:54Z

core/src/main/java/com/scalar/db/util/groupcommit/GroupManager.java

+  private final GroupCommitConfig config;
+
+  GroupManager(
+      String label,


Nit: It loos like this label is never used.

Fixed it in 0007ba3

brfrn169 · 2024-04-02T07:44:22Z

core/src/main/java/com/scalar/db/util/groupcommit/KeyManipulator.java

+
+  FULL_KEY fullKey(PARENT_KEY parentKey, CHILD_KEY childKey);
+
+  boolean isFullKey(Object obj);


Nit: It looks like it's never used.

Ah, right. Indeed, it's not used in this PR. But, it'll be used in the subsequent PR https://github.com/scalar-labs/scalardb/pull/1523/files#diff-8049e66d11b81cdd5a1dc82c4977b5336741bcee24dd7ddd54c6a6940f09497bR75. Let me keep it if possible.

brfrn169 · 2024-04-02T07:46:12Z

core/src/main/java/com/scalar/db/util/groupcommit/ThrowableRunnable.java

+package com.scalar.db.util.groupcommit;
+
+@FunctionalInterface
+interface ThrowableRunnable {


We already have ThrowableRunnable:
https://github.com/scalar-labs/scalardb/blob/53af316e1badb9460652ef7fcbb54612756edd3b/core/src/main/java/com/scalar/db/util/ThrowableRunnable.java

Fixed it in e8faee2

feeblefakie

Overall, looking good!
Left one question. PTAL!

feeblefakie · 2024-04-01T23:53:47Z

core/src/main/java/com/scalar/db/util/groupcommit/BackgroundWorker.java

+        T removed = queue.poll();
+        // Check if the removed group is expected just in case.
+        if (removed == null || !removed.equals(item)) {
+          logger.error(


I don't fully understand it, but I'm wondering in what cases this happens.
If it happens even in normal cases, should it be warning instead of error?

Good point. It shouldn't happen in normal cases. Throwing AssertionError or IllegalStateException is an option. But throwing either of them from background worker doesn't make sense so much and I think just outputting error log is enough.

But, I noticed the error message probably looks unclear thanks to your comment. I updated it in bdf344f

Looking good!
Yeah, it cannot stop processing even such a case, so having the updated comment makes sense.

komamitsu · 2024-04-05T08:14:24Z

so it might be worth trying TPC-C to see how it goes since there are not many conflicts in YCSB.

@feeblefakie Yeah, good point. I'll try TPC-C with the clustering key version group commit before submitting a PR for the integration of the group commit with the coordinator state commit.

and add some comments

komamitsu · 2024-04-05T09:54:47Z

@feeblefakie @brfrn169 Thanks for reviewing this PR!
This is a small FYI, but I renamed asyncEmit() to delegateEmitTaskToWaiter() in 545ad20 since the old method name was misleading (actually, the emit task is executed by one of the slots in the group synchronously in waitUntilEmit())

@brfrn169 The performance improvement task of DelayedSlotMoveWorker will be handled in another PR 545ad20#diff-21491e996128b6ee7175154186a58be20ca714274559ba304eb39c59be8db4faR110-R113

brfrn169

Left very minor suggestions. It looks good to me, but let me review this once again later.

brfrn169 · 2024-04-05T14:47:50Z

core/src/main/java/com/scalar/db/util/groupcommit/GroupManager.java

+    long stamp = lock.writeLock();
+    try {
+      // TODO: NormalGroup.removeNotReadySlots() calls updateStatus() potentially resulting in
+      //       asyncEmit(). Maybe it should be called outside the lock.


Nit:

Suggested change

// asyncEmit(). Maybe it should be called outside the lock.

// delegateEmitTaskToWaiter(). Maybe it should be called outside the lock.

Thanks! Fixed it in 8658300

brfrn169 · 2024-04-05T15:20:42Z

core/src/main/java/com/scalar/db/util/groupcommit/Slot.java

+  private final CHILD_KEY key;
+  // If a result value is null, the value is already emitted.
+  // Otherwise, the result lambda must be emitted by the receiver's thread.
+  private final CompletableFuture<ThrowableRunnable> completableFuture = new CompletableFuture<>();


Suggested change

private final CompletableFuture<ThrowableRunnable> completableFuture = new CompletableFuture<>();

private final CompletableFuture<ThrowableRunnable<Exception>> completableFuture = new CompletableFuture<>();

Thanks! Fixed it in 8658300

brfrn169 · 2024-04-05T15:20:54Z

core/src/main/java/com/scalar/db/util/groupcommit/Slot.java

+    try {
+      // If a result value is null, the value is already emitted.
+      // Otherwise, the result lambda must be emitted by the receiver's thread.
+      ThrowableRunnable taskToEmit = completableFuture.get();


Suggested change

ThrowableRunnable taskToEmit = completableFuture.get();

ThrowableRunnable<Exception> taskToEmit = completableFuture.get();

brfrn169 · 2024-04-05T15:21:06Z

core/src/main/java/com/scalar/db/util/groupcommit/Slot.java

+      }
+      throw new GroupCommitException(
+          String.format("Group commit failed. Group: %s", parentGroup.get()), cause);
+    } catch (Throwable e) {


Suggested change

} catch (Throwable e) {

} catch (Exception e) {

brfrn169 · 2024-04-05T15:21:18Z

core/src/main/java/com/scalar/db/util/groupcommit/Slot.java

+
+  // Delegates the emit task to the client. The client receiving this task needs to handle the emit
+  // task.
+  void delegateTaskToWaiter(ThrowableRunnable task) {


Suggested change

void delegateTaskToWaiter(ThrowableRunnable task) {

void delegateTaskToWaiter(ThrowableRunnable<Exception> task) {

Thanks! Fixed it in 8658300

brfrn169 · 2024-04-05T15:22:05Z

core/src/main/java/com/scalar/db/util/groupcommit/NormalGroup.java

+
+    // This task is passed to only the first slot, so the slot will be resumed.
+    // Other slots will be blocked until `markAsXxxx()` is called.
+    ThrowableRunnable<GroupCommitException> taskForEmitterSlot =


Suggested change

ThrowableRunnable<GroupCommitException> taskForEmitterSlot =

ThrowableRunnable<Exception> taskForEmitterSlot =

Thanks! Fixed it in 8658300

feeblefakie

LGTM! Thank you!

brfrn169

LGTM! Thank you for your great work!

Add util/groupcommit package

5c65b4d

komamitsu requested review from brfrn169, feeblefakie and Torch3333 March 14, 2024 10:54

Add missing dependency

d13199f

brfrn169 assigned komamitsu Mar 15, 2024

brfrn169 added the enhancement New feature or request label Mar 15, 2024

komamitsu added 3 commits March 15, 2024 10:45

Remove GroupCommitterBench

84c8827

Minor improvements

1e634ed

[skip ci] Add comments

165cb3d

brfrn169 reviewed Mar 22, 2024

View reviewed changes

komamitsu added 6 commits March 22, 2024 17:59

[skip ci] Tweak log level of the monitoring

ade2864

[skip ci] Make the comment clearer

fa5e726

[skip ci] Throw an exception when it's unexpected

8d96fd4

[skip ci] Update comment for remove()

4ff928a

[skip ci] Put @threadsafe annotation

e581918

and simplify some code

komamitsu requested a review from brfrn169 March 26, 2024 03:00

komamitsu added 2 commits March 27, 2024 14:48

[skip ci] Update messages to be consistent

06e14f2

[skip ci] Use consistent exception

4835ff1

feeblefakie reviewed Mar 27, 2024

View reviewed changes

komamitsu added 3 commits March 28, 2024 16:05

[skip ci] Rename CLOSED to SIZE-FIXED in Group

5000335

[skip ci] Add old-group-abort-timeout

a2e9c54

[skip ci] Stop calling GroupCommitter.remove()

53af316

in GroupCommitterConcurrentTest to see if the timeout mechanisms work

brfrn169 reviewed Apr 2, 2024

View reviewed changes

feeblefakie reviewed Apr 2, 2024

View reviewed changes

[skip ci] Remove unused variable

0007ba3

[skip ci] Refactoring

545ad20

and add some comments

brfrn169 reviewed Apr 5, 2024

View reviewed changes

[skip ci] Fix wrong comment and generics type

8658300

komamitsu requested review from brfrn169 and feeblefakie April 8, 2024 02:01

feeblefakie approved these changes Apr 8, 2024

View reviewed changes

komamitsu added 3 commits April 8, 2024 12:54

[skip ci] Add comment and test for old-group-T/O

2cc7a0d

[skip ci] Prevent potential inconsistent state

a5cbee6

[skip ci] Fix flaky test

96e5269

brfrn169 approved these changes Apr 8, 2024

View reviewed changes

Merge branch 'master' into add-group-commit-package

ab2891d

brfrn169 merged commit cca9421 into master Apr 9, 2024
23 checks passed

brfrn169 deleted the add-group-commit-package branch April 9, 2024 00:30

feeblefakie pushed a commit that referenced this pull request Apr 15, 2024

Add util/groupcommit package in core subproject (#1607)

4e8107c

feeblefakie mentioned this pull request Apr 15, 2024

Backport to branch(3.12) : Add util/groupcommit package in core subproject #1666

Closed

feeblefakie pushed a commit that referenced this pull request Apr 15, 2024

Add util/groupcommit package in core subproject (#1607)

a98ad89

feeblefakie mentioned this pull request Apr 15, 2024

Backport to branch(3.11) : Add util/groupcommit package in core subproject #1667

Closed

feeblefakie pushed a commit that referenced this pull request Apr 15, 2024

Add util/groupcommit package in core subproject (#1607)

9cf3a99

feeblefakie mentioned this pull request Apr 15, 2024

Backport to branch(3.10) : Add util/groupcommit package in core subproject #1668

Closed

feeblefakie pushed a commit that referenced this pull request Apr 15, 2024

Add util/groupcommit package in core subproject (#1607)

90b804c

feeblefakie mentioned this pull request Apr 15, 2024

Backport to branch(3.9) : Add util/groupcommit package in core subproject #1669

Closed

feeblefakie pushed a commit that referenced this pull request Apr 15, 2024

Add util/groupcommit package in core subproject (#1607)

94d18b2

feeblefakie mentioned this pull request Apr 15, 2024

Backport to branch(3.8) : Add util/groupcommit package in core subproject #1670

Closed

feeblefakie pushed a commit that referenced this pull request Apr 15, 2024

Add util/groupcommit package in core subproject (#1607)

ebac166

feeblefakie mentioned this pull request Apr 15, 2024

Backport to branch(3) : Add util/groupcommit package in core subproject #1671

Merged

komamitsu mentioned this pull request Apr 19, 2024

Make group commit metrics log configurable #1684

Merged

6 tasks

komamitsu mentioned this pull request May 15, 2024

Integrate util/groupcommit package with the coordinator table commit #1728

Merged

6 tasks


		FULL_KEY fullKey(PARENT_KEY parentKey, CHILD_KEY childKey);

		boolean isFullKey(Object obj);

	// asyncEmit(). Maybe it should be called outside the lock.
	// delegateEmitTaskToWaiter(). Maybe it should be called outside the lock.

	private final CompletableFuture<ThrowableRunnable> completableFuture = new CompletableFuture<>();
	private final CompletableFuture<ThrowableRunnable<Exception>> completableFuture = new CompletableFuture<>();

	ThrowableRunnable taskToEmit = completableFuture.get();
	ThrowableRunnable<Exception> taskToEmit = completableFuture.get();

	void delegateTaskToWaiter(ThrowableRunnable task) {
	void delegateTaskToWaiter(ThrowableRunnable<Exception> task) {

	ThrowableRunnable<GroupCommitException> taskForEmitterSlot =
	ThrowableRunnable<Exception> taskForEmitterSlot =

Add util/groupcommit package in core subproject #1607

Add util/groupcommit package in core subproject #1607

Conversation

komamitsu commented Mar 14, 2024 • edited Loading

Description

Related issues and/or PRs

Changes made

Checklist

Additional notes (optional)

Release notes

brfrn169 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feeblefakie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

komamitsu commented Mar 28, 2024

feeblefakie commented Apr 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feeblefakie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

komamitsu commented Apr 5, 2024

komamitsu commented Apr 5, 2024

brfrn169 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feeblefakie left a comment

Choose a reason for hiding this comment

brfrn169 left a comment

Choose a reason for hiding this comment

Add `util/groupcommit` package in `core` subproject #1607

Add `util/groupcommit` package in `core` subproject #1607

komamitsu commented Mar 14, 2024 •

edited

Loading