No tracing memory usage of shared column data in MPPTask's memory tracker by JinheLin · Pull Request #8131 · pingcap/tiflash

JinheLin · 2023-09-21T11:09:16Z

What problem does this PR solve?

Issue Number: close #8128

What is changed and how it works?

In DMFileReader::readCoumn, if there are some concurrent DMFileReader of the same file existing, this column data is likely to be shared, so disable memory tracing by setting current_memory_tracker to nullptr.
Mark PODArray the underlying memory of column data to be is_shared_memory and use shared_column_data_mem_tracker to trace its memory.

Before this PR(from issue #8128)

[2023/09/21 09:08:37.087 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 10.48 GiB."] [source=MemoryTracker] [thread_id=269]
[2023/09/21 09:08:37.090 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 288.86 MiB."] [source=MemoryTracker] [thread_id=270]
[2023/09/21 09:08:37.090 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (total): 988.18 MiB."] [source=MemoryTracker] [thread_id=270]
[2023/09/21 09:08:38.407 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 8.28 GiB."] [source=MemoryTracker] [thread_id=275]
[2023/09/21 09:08:38.824 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 244.48 MiB."] [source=MemoryTracker] [thread_id=277]
[2023/09/21 09:08:39.801 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 3.79 GiB."] [source=MemoryTracker] [thread_id=280]
[2023/09/21 09:08:40.381 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 288.98 MiB."] [source=MemoryTracker] [thread_id=283]
[2023/09/21 09:08:41.408 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 2.69 GiB."] [source=MemoryTracker] [thread_id=286]
[2023/09/21 09:08:42.029 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 397.90 MiB."] [source=MemoryTracker] [thread_id=289]
[2023/09/21 09:08:43.009 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 4.41 GiB."] [source=MemoryTracker] [thread_id=292]
[2023/09/21 09:08:43.488 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (for query): 435.62 MiB."] [source=MemoryTracker] [thread_id=295]
[2023/09/21 09:08:43.488 +08:00] [DEBUG] [MemoryTracker.cpp:101] ["Peak memory usage (total): 8.89 GiB."] [source=MemoryTracker] [thread_id=295]

After this PR:

[2023/09/22 17:20:49.967 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.04 GiB."] [source=MemoryTracker] [thread_id=3084]
[2023/09/22 17:20:53.023 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.60 GiB."] [source=MemoryTracker] [thread_id=3086]
[2023/09/22 17:20:53.132 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.07 GiB."] [source=MemoryTracker] [thread_id=3089]
[2023/09/22 17:20:56.188 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.21 GiB."] [source=MemoryTracker] [thread_id=3091]
[2023/09/22 17:20:56.298 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.02 GiB."] [source=MemoryTracker] [thread_id=3094]
[2023/09/22 17:20:59.324 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.43 GiB."] [source=MemoryTracker] [thread_id=3096]
[2023/09/22 17:20:59.440 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.03 GiB."] [source=MemoryTracker] [thread_id=3098]
[2023/09/22 17:21:02.469 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.28 GiB."] [source=MemoryTracker] [thread_id=3100]
[2023/09/22 17:21:02.564 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.03 GiB."] [source=MemoryTracker] [thread_id=3103]
[2023/09/22 17:21:05.894 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.38 GiB."] [source=MemoryTracker] [thread_id=3105]
[2023/09/22 17:21:05.991 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.01 GiB."] [source=MemoryTracker] [thread_id=3108]
[2023/09/22 17:21:09.070 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.69 GiB."] [source=MemoryTracker] [thread_id=3109]
[2023/09/22 17:21:09.187 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.30 GiB."] [source=MemoryTracker] [thread_id=3112]
[2023/09/22 17:21:12.246 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.25 GiB."] [source=MemoryTracker] [thread_id=3114]
[2023/09/22 17:21:12.361 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.11 GiB."] [source=MemoryTracker] [thread_id=3118]
[2023/09/22 17:21:15.412 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.30 GiB."] [source=MemoryTracker] [thread_id=3120]
[2023/09/22 17:21:15.569 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.04 GiB."] [source=MemoryTracker] [thread_id=3122]
[2023/09/22 17:21:18.653 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.33 GiB."] [source=MemoryTracker] [thread_id=3125]
[2023/09/22 17:21:18.769 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.71 GiB."] [source=MemoryTracker] [thread_id=3128]
[2023/09/22 17:21:21.845 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.28 GiB."] [source=MemoryTracker] [thread_id=3131]
[2023/09/22 17:21:21.988 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.08 GiB."] [source=MemoryTracker] [thread_id=3134]
[2023/09/22 17:21:25.001 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.35 GiB."] [source=MemoryTracker] [thread_id=3137]
[2023/09/22 17:21:25.135 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.05 GiB."] [source=MemoryTracker] [thread_id=3139]
[2023/09/22 17:21:28.144 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.22 GiB."] [source=MemoryTracker] [thread_id=3142]
[2023/09/22 17:21:28.286 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.12 GiB."] [source=MemoryTracker] [thread_id=3143]
[2023/09/22 17:21:31.336 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.20 GiB."] [source=MemoryTracker] [thread_id=3144]
[2023/09/22 17:21:31.454 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.32 GiB."] [source=MemoryTracker] [thread_id=3145]
[2023/09/22 17:21:34.535 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.28 GiB."] [source=MemoryTracker] [thread_id=3146]
[2023/09/22 17:21:34.650 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1023.65 MiB."] [source=MemoryTracker] [thread_id=3148]
[2023/09/22 17:21:37.665 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.31 GiB."] [source=MemoryTracker] [thread_id=3151]
[2023/09/22 17:21:37.799 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.09 GiB."] [source=MemoryTracker] [thread_id=3153]
[2023/09/22 17:21:40.931 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.28 GiB."] [source=MemoryTracker] [thread_id=3155]
[2023/09/22 17:21:41.051 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.20 GiB."] [source=MemoryTracker] [thread_id=3157]
[2023/09/22 17:21:44.169 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.24 GiB."] [source=MemoryTracker] [thread_id=3159]
[2023/09/22 17:21:44.294 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.10 GiB."] [source=MemoryTracker] [thread_id=3162]
[2023/09/22 17:21:47.329 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.25 GiB."] [source=MemoryTracker] [thread_id=3165]
[2023/09/22 17:21:47.441 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.12 GiB."] [source=MemoryTracker] [thread_id=3167]
[2023/09/22 17:21:50.481 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.53 GiB."] [source=MemoryTracker] [thread_id=3170]
[2023/09/22 17:21:50.657 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 998.26 MiB."] [source=MemoryTracker] [thread_id=3171]
[2023/09/22 17:21:53.689 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.67 GiB."] [source=MemoryTracker] [thread_id=3173]
[2023/09/22 17:21:53.801 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.88 GiB."] [source=MemoryTracker] [thread_id=3176]
[2023/09/22 17:21:56.932 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.36 GiB."] [source=MemoryTracker] [thread_id=3179]
[2023/09/22 17:21:57.026 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.06 GiB."] [source=MemoryTracker] [thread_id=3181]
[2023/09/22 17:22:00.085 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.62 GiB."] [source=MemoryTracker] [thread_id=3183]
[2023/09/22 17:22:00.210 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.13 GiB."] [source=MemoryTracker] [thread_id=3185]
[2023/09/22 17:22:03.338 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.35 GiB."] [source=MemoryTracker] [thread_id=3186]
[2023/09/22 17:22:03.421 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.06 GiB."] [source=MemoryTracker] [thread_id=3188]
[2023/09/22 17:22:06.471 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.48 GiB."] [source=MemoryTracker] [thread_id=3191]
[2023/09/22 17:22:06.589 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (for query): 1.05 GiB."] [source=MemoryTracker] [thread_id=3193]
[2023/09/22 17:22:06.589 +08:00] [DEBUG] [MemoryTracker.cpp:107] ["Peak memory usage (total): 3.86 GiB."] [source=MemoryTracker] [thread_id=3193]

Maybe there are some global PODArray object that are not being traced before, about 12.4MiB.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Fix the issue that memory usage tracking is not accurate

ti-chi-bot · 2023-09-21T11:09:25Z

Hi @JinheLin. Thanks for your PR.

I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

JinheLin · 2023-09-21T11:14:39Z

@flowbehappy @windtalker @Lloyd-Pottiger @JaySon-Huang PTAL.

JinheLin · 2023-09-22T01:05:58Z

/run-all-tests

dbms/src/Common/PODArray.h

dbms/src/Storages/DeltaMerge/ReadThread/ColumnSharingCache.h

…cker.

JinheLin · 2023-09-22T04:58:08Z

/run-all-tests

JinheLin · 2023-09-22T05:46:06Z

/run-integration-test

JinheLin · 2023-09-22T06:05:17Z

/run-all-tests

JinheLin · 2023-09-22T07:09:15Z

/run-unit-test

JaySon-Huang

Rest LGTM

dbms/src/Common/MemoryTracker.cpp

Co-authored-by: JaySon <tshent@qq.com>

JinheLin · 2023-09-22T07:41:02Z

/run-all-tests

JaySon-Huang

LGTM

windtalker · 2023-09-25T01:31:35Z

dbms/src/Common/MemoryTracker.cpp

    if (!next.load(std::memory_order_relaxed))
+    {
        CurrentMetrics::add(metric, size);
+        if (shared_column_data_mem_tracker)


Maybe data race here?

shared_column_data_mem_tracker is a global object and has initialized when server starting. I think there is no data race on it after its initialization completed.

Yes, I aggree, but theoretically there will be data race here, not sure if our ASan tests will complain it or not. I'm ok with current implementation, and I think we can fix it if it breaks the ASan tests.

windtalker · 2023-09-25T01:34:57Z

dbms/src/Storages/DeltaMerge/File/DMFileReader.cpp

+        // So the allocation and deallocation of this data may not be in the same MemoryTracker.
+        // This can lead to inaccurate memory statistics of MemoryTracker.
+        // To solve this problem, we use a independent global memory tracker to trace the shared column data in ColumnSharingCacheMap.
+        MemoryTrackerSetter mem_tracker_guard(true, has_concurrent_reader ? nullptr : current_memory_tracker);


Don't have to use this mem_tracker_guard if has_concurrent_reader is false?

dbms/src/Storages/DeltaMerge/ReadThread/ColumnSharingCache.cpp

windtalker

LGTM

windtalker · 2023-09-25T02:33:38Z

dbms/src/Common/MemoryTracker.cpp

    if (!next.load(std::memory_order_relaxed))
+    {
        CurrentMetrics::add(metric, size);
+        if (shared_column_data_mem_tracker)


Yes, I aggree, but theoretically there will be data race here, not sure if our ASan tests will complain it or not. I'm ok with current implementation, and I think we can fix it if it breaks the ASan tests.

ti-chi-bot · 2023-09-25T02:33:53Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JaySon-Huang, windtalker

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [JaySon-Huang,windtalker]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2023-09-25T02:33:55Z

[LGTM Timeline notifier]

Timeline:

2023-09-22 08:10:29.115853539 +0000 UTC m=+3218.834195749: ☑️ agreed by JaySon-Huang.
2023-09-25 02:33:54.596139336 +0000 UTC m=+242224.314481552: ☑️ agreed by windtalker.

JinheLin · 2023-09-25T02:43:53Z

/run-all-tests

ti-chi-bot · 2023-09-25T02:44:08Z

@JinheLin: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

trigger some heavy tests which will not run always when PR updated.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

JaySon-Huang · 2023-09-25T03:51:15Z

/run-unit-test

ti-chi-bot · 2023-09-25T04:27:54Z

In response to a cherrypick label: new pull request created to branch release-6.5: #8135.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2023-09-25T04:28:32Z

In response to a cherrypick label: new pull request created to branch release-7.1: #8136.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

…cker (#8131) (#8136) close #8128

…cker (#8131) (#8135) close #8128

windtalker reviewed Sep 22, 2023

View reviewed changes

dbms/src/Common/PODArray.h Outdated Show resolved Hide resolved

dbms/src/Storages/DeltaMerge/ReadThread/ColumnSharingCache.h Outdated Show resolved Hide resolved

dbms/src/Storages/DeltaMerge/ReadThread/ColumnSharingCache.h Outdated Show resolved Hide resolved

JinheLin marked this pull request as draft September 22, 2023 02:43

ti-chi-bot bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 22, 2023

JinheLin force-pushed the fix-shared-column branch from a30c5c9 to ef1801d Compare September 22, 2023 03:27

JinheLin added 2 commits September 22, 2023 11:27

No tracing memory usage of shared column data in MPPTask's memory tra…

2ba70b5

…cker.

ci

a1bb74a

JinheLin force-pushed the fix-shared-column branch from ef1801d to a1bb74a Compare September 22, 2023 03:29

JinheLin added 2 commits September 22, 2023 11:47

ci

48b34b7

ci

105a5b0

fix ut

cbe5418

JinheLin marked this pull request as ready for review September 22, 2023 06:25

ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 22, 2023

JinheLin requested a review from windtalker September 22, 2023 06:47

JaySon-Huang reviewed Sep 22, 2023

View reviewed changes

dbms/src/Common/MemoryTracker.cpp Outdated Show resolved Hide resolved

JinheLin and others added 2 commits September 22, 2023 15:40

fix ut

198d22d

Update dbms/src/Common/MemoryTracker.cpp

38b310f

Co-authored-by: JaySon <tshent@qq.com>

JaySon-Huang approved these changes Sep 22, 2023

View reviewed changes

ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Sep 22, 2023

windtalker reviewed Sep 25, 2023

View reviewed changes

dbms/src/Storages/DeltaMerge/ReadThread/ColumnSharingCache.cpp Show resolved Hide resolved

MemoryTracker optional.

3bc1dec

windtalker approved these changes Sep 25, 2023

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Sep 25, 2023

Merge branch 'master' into fix-shared-column

99dae27

ti-chi-bot bot merged commit a7cdb82 into pingcap:master Sep 25, 2023

ti-chi-bot mentioned this pull request Sep 25, 2023

No tracing memory usage of shared column data in MPPTask's memory tracker (#8131) #8135

Merged

12 tasks

ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Sep 25, 2023

This is an automated cherry-pick of pingcap#8131

1d44b88

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Sep 25, 2023

No tracing memory usage of shared column data in MPPTask's memory tracker (#8131) #8136

Merged

12 tasks

ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Sep 25, 2023

This is an automated cherry-pick of pingcap#8131

8f3261b

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot bot pushed a commit that referenced this pull request Sep 25, 2023

No tracing memory usage of shared column data in MPPTask's memory tra…

1b60452

…cker (#8131) (#8136) close #8128

ti-chi-bot bot pushed a commit that referenced this pull request Sep 27, 2023

No tracing memory usage of shared column data in MPPTask's memory tra…

9dca4d2

…cker (#8131) (#8135) close #8128

ti-chi-bot bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Oct 18, 2023

Conversation

JinheLin commented Sep 21, 2023 • edited by JaySon-Huang Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

Uh oh!

ti-chi-bot bot commented Sep 21, 2023

Uh oh!

JinheLin commented Sep 21, 2023

Uh oh!

JinheLin commented Sep 22, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JinheLin commented Sep 22, 2023

Uh oh!

JinheLin commented Sep 22, 2023

Uh oh!

JinheLin commented Sep 22, 2023

Uh oh!

JinheLin commented Sep 22, 2023

Uh oh!

JaySon-Huang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

JinheLin commented Sep 22, 2023

Uh oh!

JaySon-Huang left a comment

Choose a reason for hiding this comment

Uh oh!

windtalker Sep 25, 2023

Choose a reason for hiding this comment

Uh oh!

JinheLin Sep 25, 2023

Choose a reason for hiding this comment

Uh oh!

windtalker Sep 25, 2023

Choose a reason for hiding this comment

Uh oh!

windtalker Sep 25, 2023

Choose a reason for hiding this comment

Uh oh!

JinheLin Sep 25, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

windtalker left a comment

Choose a reason for hiding this comment

Uh oh!

windtalker Sep 25, 2023

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented Sep 25, 2023

Uh oh!

ti-chi-bot bot commented Sep 25, 2023

[LGTM Timeline notifier]

Uh oh!

JinheLin commented Sep 25, 2023

Uh oh!

ti-chi-bot bot commented Sep 25, 2023

Uh oh!

JaySon-Huang commented Sep 25, 2023

Uh oh!

ti-chi-bot commented Sep 25, 2023

Uh oh!

ti-chi-bot commented Sep 25, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

JinheLin commented Sep 21, 2023 •

edited by JaySon-Huang

Loading