karmada-scheduler add disable-scheduler-estimator-in-pull-mode flag #2064

prodanlabs · 2022-06-25T10:08:17Z

Signed-off-by: prodan pengshihaoren@gmail.com

What type of PR is this?

/kind bug
/kind feature

What this PR does / why we need it:

In pull mode, if the network of the member cluster and karmada are not two-way communication， scheduler-estimator is not available.，In this scenario, we need to add a flag to disable scheduler-estimator in pull mode.

For example, my cluster2 is in pull mode, and scheduler-estimator is not installed, the scheduler has been looking for karmada-scheduler-estimator-cluster2.

root@dev-karmada-cluster02:~# kubectl  --kubeconfig /etc/karmada/karmada-apiserver.config get cluster
NAME       VERSION   MODE   READY   AGE
cluster1   v1.23.8   Push   True    72m
cluster2   v1.22.0   Pull   True    69m

I0625 09:19:38.584300       1 cache.go:87] Start dialing estimator server(karmada-scheduler-estimator-cluster2:10352) of cluster(cluster2).
W0625 09:19:38.587111       1 clientconn.go:1322] [core] grpc: addrConn.createTransport failed to connect to {karmada-scheduler-estimator-cluster2:10352 karmada-scheduler-estimator-cluster2:10352 <nil> <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup karmada-scheduler-estimator-cluster2 on 10.96.0.10:53: no such host"
W0625 09:19:39.590520       1 clientconn.go:1322] [core] grpc: addrConn.createTransport failed to connect to {karmada-scheduler-estimator-cluster2:10352 karmada-scheduler-estimator-cluster2:10352 <nil> <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup karmada-scheduler-estimator-cluster2 on 10.96.0.10:53: no such host"
W0625 09:19:41.431697       1 clientconn.go:1322] [core] grpc: addrConn.createTransport failed to connect to {karmada-scheduler-estimator-cluster2:10352 karmada-scheduler-estimator-cluster2:10352 <nil> <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup karmada-scheduler-estimator-cluster2 on 10.96.0.10:53: no such host"
E0625 09:19:43.584807       1 cache.go:90] Failed to dial cluster(cluster2): dial karmada-scheduler-estimator-cluster2:10352 error: context deadline exceeded.
I0625 09:19:43.745008       1 cache.go:87] Start dialing estimator server(karmada-scheduler-estimator-cluster2:10352) of cluster(cluster2).
W0625 09:19:43.748414       1 clientconn.go:1322] [core] grpc: addrConn.createTransport failed to connect to {karmada-scheduler-estimator-cluster2:10352 karmada-scheduler-estimator-cluster2:10352 <nil> <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup karmada-scheduler-estimator-cluster2 on 10.96.0.10:53: no such host"
W0625 09:19:44.752193       1 clientconn.go:1322] [core] grpc: addrConn.createTransport failed to connect to {karmada-scheduler-estimator-cluster2:10352 karmada-scheduler-estimator-cluster2:10352 <nil> <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup karmada-scheduler-estimator-cluster2 on 10.96.0.10:53: no such host"
W0625 09:19:46.094775       1 clientconn.go:1322] [core] grpc: addrConn.createTransport failed to connect to {karmada-scheduler-estimator-cluster2:10352 karmada-scheduler-estimator-cluster2:10352 <nil> <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup karmada-scheduler-estimator-cluster2 on 10.96.0.10:53: no such host"
W0625 09:19:48.258426       1 clientconn.go:1322] [core] grpc: addrConn.createTransport failed to connect to {karmada-scheduler-estimator-cluster2:10352 karmada-scheduler-estimator-cluster2:10352 <nil> <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup karmada-scheduler-estimator-cluster2 on 10.96.0.10:53: no such host"
E0625 09:19:48.745915       1 cache.go:90] Failed to dial cluster(cluster2): dial karmada-scheduler-estimator-cluster2:10352 error: context deadline exceeded.
I0625 09:19:49.066524       1 cache.go:87] Start dialing estimator server(karmada-scheduler-estimator-cluster2:10352) of cluster(cluster2).
W0625 09:19:49.069144       1 clientconn.go:1322] [core] grpc: addrConn.createTransport failed to connect to {karmada-scheduler-estimator-cluster2:10352 karmada-scheduler-estimator-cluster2:10352 <nil> <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup karmada-scheduler-estimator-cluster2 on 10.96.0.10:53: no such host"
W0625 09:19:50.072502       1 clientconn.go:1322] [core] grpc: addrConn.createTransport failed to connect to {karmada-scheduler-estimator-cluster2:10352 karmada-scheduler-estimator-cluster2:10352 <nil> <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup karmada-scheduler-estimator-cluster2 on 10.96.0.10:53: no such host"
W0625 09:19:51.583913       1 clientconn.go:1322] [core] grpc: addrConn.createTransport failed to connect to {karmada-scheduler-estimator-cluster2:10352 karmada-scheduler-estimator-cluster2:10352 <nil> <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup karmada-scheduler-estimator-cluster2 on 10.96.0.10:53: no such host"
E0625 09:19:54.066914       1 cache.go:90] Failed to dial cluster(cluster2): dial karmada-scheduler-estimator-cluster2:10352 error: context deadline exceeded.
I0625 09:19:54.707938       1 cache.go:87] Start dialing estimator server(karmada-scheduler-estimator-cluster2:10352) of cluster(cluster2).
W0625 09:19:54.710377       1 clientconn.go:1322] [core] grpc: addrConn.createTransport failed to connect to {karmada-scheduler-estimator-cluster2:10352 karmada-scheduler-estimator-cluster2:10352 <nil> <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup karmada-scheduler-estimator-cluster2 on 10.96.0.10:53: no such host"
W0625 09:19:55.713421       1 clientconn.go:1322] [core] grpc: addrConn.createTransport failed to connect to {karmada-scheduler-estimator-cluster2:10352 karmada-scheduler-estimator-cluster2:10352 <nil> <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup karmada-scheduler-estimator-cluster2 on 10.96.0.10:53: no such host"
W0625 09:19:57.567846       1 clientconn.go:1322] [core] grpc: addrConn.createTransport failed to connect to {karmada-scheduler-estimator-cluster2:10352 karmada-scheduler-estimator-cluster2:10352 <nil> <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup karmada-scheduler-estimator-cluster2 on 10.96.0.10:53: no such host"
E0625 09:19:59.708969       1 cache.go:90] Failed to dial cluster(cluster2): dial karmada-scheduler-estimator-cluster2:10352 error: context deadline exceeded.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

`karmada-scheduler`: Introduced  `--disable-scheduler-estimator-in-pull-mode` flag to disable scheduler-estimator for pull mode clusters.

Garrybest · 2022-06-25T15:07:30Z

Actually scheduler-estimator is also used for pulled clusters. If you enable scheduler estimator option in karmada-scheduler, it will look for all estimators. Refer to

karmada/docs/scheduler-estimator.md

Line 33 in 59219f4

    
           - If the cluster has been joined already, you could use `hack/deploy-scheduler-estimator.sh` to only deploy karmada-scheduler-estimator.

prodanlabs · 2022-06-25T22:23:51Z

Actually scheduler-estimator is also used for pulled clusters. If you enable scheduler estimator option in karmada-scheduler, it will look for all estimators. Refer to

karmada/docs/scheduler-estimator.md

Line 33 in 59219f4

- If the cluster has been joined already, you could use `hack/deploy-scheduler-estimator.sh` to only deploy karmada-scheduler-estimator.

thank you for your reply.

I think the pull mode is used to solve the one-way network problem between the member cluster and karmada, so deploy scheduler-estimator in the host cluster, scheduler-estimator cannot connect to the kube-apiserver that connects to the member cluster.

In the case where the network can communicate in both directions, karmada should recommend users to use the push mode, because from the details, the push mode is more perfect.

Our private cloud's kube-apiserver is not open to the public network, including proxies, for security reasons，so scheduler-estimator is useless for our pull mode. FYR

Garrybest · 2022-06-26T02:06:41Z

Hi @prodanlabs, I got you. But some users choose the pull mode not for network reasons. AFAIK, just for HA or some other performance reasons. So I think we should try to use a tunnel or proxy to fix the network barrier, it may be not appropriate to just disable scheduler-estimator in pull mode. How do you think?

prodanlabs · 2022-06-26T03:08:43Z

@Garrybest Thanks for the quick response

Yes, I admit that some users still use pull mode even when the network can communicate in both directions.

So I think we should try to use a tunnel or proxy to fix the network barrier

In fact, karmada also provides anp solutions to solve network obstacles. In pull mode, karmada-kubectl logs, exec and other subcommands can also get logs or enter Pod.

But as far as we are concerned, because of security management regulations, the core business systems on the private cloud are not allowed to be accessed through the public network, nor can network agents or network tunnels be deployed.

My idea is that push mode is recommended when there is no network failure. Pull mode does not need to deploy scheduler estimator, or compromise, in pull mode, scheduler adds a flag to close scheduler estimator event.

Garrybest · 2022-06-27T01:07:09Z

It makes sense. We could add a flag to avoid establishing the connection with clusters in pull mode. Would you like to revise again?

prodanlabs · 2022-06-27T01:59:20Z

It makes sense. We could add a flag to avoid establishing the connection with clusters in pull mode. Would you like to revise again?

OK

prodanlabs · 2022-06-27T02:01:51Z

Hi @Garrybest , do you have any good suggestions for flag names?

prodanlabs · 2022-06-27T05:39:07Z

@Garrybest @RainbowMango can you please review?

Garrybest · 2022-06-27T07:33:37Z

Hi @prodanlabs, this flag is not just used for disable event. If we don't add pull-mode clusters into schedulerEstimatorWorker, the scheduler will never establish a connection with them. So the flag is to disable scheduler estimator in pull-mode clusters.

The option could be like DisableSchedulerEstimatorInPullMode.

prodanlabs · 2022-06-27T07:47:22Z

Hi @prodanlabs, this flag is not just used for disable event. If we don't add pull-mode clusters into schedulerEstimatorWorker, the scheduler will never establish a connection with them. So the flag is to disable scheduler estimator in pull-mode clusters.

The option could be like DisableSchedulerEstimatorInPullMode.

Agreed, I'll change it later. thanks

prodanlabs · 2022-06-27T12:47:19Z

Hi @prodanlabs, this flag is not just used for disable event. If we don't add pull-mode clusters into schedulerEstimatorWorker, the scheduler will never establish a connection with them. So the flag is to disable scheduler estimator in pull-mode clusters.

The option could be like DisableSchedulerEstimatorInPullMode.

done.

/cc @Garrybest @RainbowMango

Garrybest · 2022-06-29T06:39:44Z

/lgtm

Thanks!

RainbowMango

Looks good to me.
/approve

/hold
For @Tingtal to confirm the grammar.
and for @prodanlabs update the PR descriptions. (Since this PR has changed a lot after discussion, so please help update the PR descriptions as well. )

RainbowMango · 2022-06-29T09:52:55Z

cmd/scheduler/app/options/options.go

@@ -80,6 +82,7 @@ func (o *Options) AddFlags(fs *pflag.FlagSet) {
 	fs.Float32Var(&o.KubeAPIQPS, "kube-api-qps", 40.0, "QPS to use while talking with karmada-apiserver. Doesn't cover events and node heartbeat apis which rate limiting is controlled by a different set of flags.")
 	fs.IntVar(&o.KubeAPIBurst, "kube-api-burst", 60, "Burst to use while talking with karmada-apiserver. Doesn't cover events and node heartbeat apis which rate limiting is controlled by a different set of flags.")
 	fs.BoolVar(&o.EnableSchedulerEstimator, "enable-scheduler-estimator", false, "Enable calling cluster scheduler estimator for adjusting replicas.")
+	fs.BoolVar(&o.DisableSchedulerEstimatorInPullMode, "disable-scheduler-estimator-in-pull-mode", false, "Disable scheduler estimator in pull mode clusters, only takes effect when enable-scheduler-estimator is true.")


Suggested change

fs.BoolVar(&o.DisableSchedulerEstimatorInPullMode, "disable-scheduler-estimator-in-pull-mode", false, "Disable scheduler estimator in pull mode clusters, only takes effect when enable-scheduler-estimator is true.")

fs.BoolVar(&o.DisableSchedulerEstimatorInPullMode, "disable-scheduler-estimator-in-pull-mode", false, "Disable scheduler estimator for pull mode clusters, only takes effect when enable-scheduler-estimator is true.")

I'm not sure if we should use for here. @Tingtal can you help to confirm?

Suggested change

fs.BoolVar(&o.DisableSchedulerEstimatorInPullMode, "disable-scheduler-estimator-in-pull-mode", false, "Disable scheduler estimator in pull mode clusters, only takes effect when enable-scheduler-estimator is true.")

fs.BoolVar(&o.DisableSchedulerEstimatorInPullMode, "disable-scheduler-estimator-in-pull-mode", false, "Disable the scheduler estimator for clusters in pull mode, which takes effect only when enable-scheduler-estimator is true.")

Tingtal

Two small touches. Pls check.

Tingtal · 2022-06-30T02:23:15Z

cmd/scheduler/app/options/options.go

@@ -80,6 +82,7 @@ func (o *Options) AddFlags(fs *pflag.FlagSet) {
 	fs.Float32Var(&o.KubeAPIQPS, "kube-api-qps", 40.0, "QPS to use while talking with karmada-apiserver. Doesn't cover events and node heartbeat apis which rate limiting is controlled by a different set of flags.")
 	fs.IntVar(&o.KubeAPIBurst, "kube-api-burst", 60, "Burst to use while talking with karmada-apiserver. Doesn't cover events and node heartbeat apis which rate limiting is controlled by a different set of flags.")
 	fs.BoolVar(&o.EnableSchedulerEstimator, "enable-scheduler-estimator", false, "Enable calling cluster scheduler estimator for adjusting replicas.")
+	fs.BoolVar(&o.DisableSchedulerEstimatorInPullMode, "disable-scheduler-estimator-in-pull-mode", false, "Disable scheduler estimator in pull mode clusters, only takes effect when enable-scheduler-estimator is true.")


Suggested change

fs.BoolVar(&o.DisableSchedulerEstimatorInPullMode, "disable-scheduler-estimator-in-pull-mode", false, "Disable scheduler estimator in pull mode clusters, only takes effect when enable-scheduler-estimator is true.")

fs.BoolVar(&o.DisableSchedulerEstimatorInPullMode, "disable-scheduler-estimator-in-pull-mode", false, "Disable the scheduler estimator for clusters in pull mode, which takes effect only when enable-scheduler-estimator is true.")

Tingtal · 2022-06-30T02:29:02Z

pkg/scheduler/scheduler.go


 	enableEmptyWorkloadPropagation bool
 }

 type schedulerOptions struct {
 	// enableSchedulerEstimator represents whether the accurate scheduler estimator should be enabled.
 	enableSchedulerEstimator bool
+	// disableSchedulerEstimatorInPullMode represents  whether to disable the pull mode scheduler estimator.


Suggested change

// disableSchedulerEstimatorInPullMode represents whether to disable the pull mode scheduler estimator.

// disableSchedulerEstimatorInPullMode represents whether to disable the scheduler estimator in pull mode.

prodanlabs · 2022-06-30T03:26:23Z

Hi @XiShanYongYe-Chang , can you help take a look.
https://github.com/karmada-io/karmada/runs/7123991413?check_suite_focus=true

XiShanYongYe-Chang · 2022-06-30T03:28:27Z

Hi @prodanlabs, have you rebased the newest code in the master branch?

prodanlabs · 2022-06-30T03:35:15Z

Hi @prodanlabs, have you rebased the newest code in the master branch?

my master branch is not up to date.

XiShanYongYe-Chang · 2022-06-30T03:38:03Z

There is a bug #2072 which we have fixed by #2074, maybe you can rebase the master branch.

prodanlabs · 2022-06-30T03:43:47Z

How to rerun CI .

XiShanYongYe-Chang · 2022-06-30T03:55:05Z

How to rerun CI .

Rebase and push again?

prodanlabs · 2022-06-30T04:09:08Z

Rebase and push again?

Last time I seemed to see @RainbowMango used the command to rerun CI, I forgot where I saw it

Signed-off-by: prodan <pengshihaoren@gmail.com>

RainbowMango · 2022-06-30T08:51:05Z

Last time I seemed to see @RainbowMango used the command to rerun CI, I forgot where I saw it

No command can be used to re-trigger the test. sad...

RainbowMango · 2022-06-30T08:52:08Z

/lgtm
/approve
/hold cancel

karmada-bot · 2022-06-30T08:52:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: RainbowMango

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [RainbowMango]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

karmada-bot added kind/bug Categorizes issue or PR as related to a bug. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. labels Jun 25, 2022

karmada-bot requested review from Garrybest and qianjun1993 June 25, 2022 10:08

karmada-bot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Jun 25, 2022

prodanlabs changed the title ~~fix: remove Scheduler estimator Event in Pull mode~~ fix: remove scheduler estimator event in pull mode Jun 25, 2022

karmada-bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jun 27, 2022

prodanlabs force-pushed the fix-scheduler branch 2 times, most recently from e75e2fc to ee3938e Compare June 27, 2022 05:25

prodanlabs changed the title ~~fix: remove scheduler estimator event in pull mode~~ karmada-scheduler add enable-pull-scheduler-estimator-event flag Jun 27, 2022

prodanlabs force-pushed the fix-scheduler branch from ee3938e to 403db45 Compare June 27, 2022 05:51

karmada-bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 27, 2022

prodanlabs force-pushed the fix-scheduler branch from 788ffa5 to a5a01dc Compare June 27, 2022 08:08

prodanlabs changed the title ~~karmada-scheduler add enable-pull-scheduler-estimator-event flag~~ karmada-scheduler add disable-scheduler-estimator-in-pull-mode flag Jun 27, 2022

karmada-bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jun 27, 2022

prodanlabs force-pushed the fix-scheduler branch from 7aac8e4 to d05fed5 Compare June 27, 2022 12:44

prodanlabs force-pushed the fix-scheduler branch 2 times, most recently from 8c9ace1 to 09358b7 Compare June 28, 2022 08:59

karmada-bot added the lgtm Indicates that a PR is ready to be merged. label Jun 29, 2022

RainbowMango approved these changes Jun 29, 2022

View reviewed changes

karmada-bot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jun 29, 2022

Tingtal suggested changes Jun 30, 2022

View reviewed changes

karmada-bot removed the lgtm Indicates that a PR is ready to be merged. label Jun 30, 2022

prodanlabs force-pushed the fix-scheduler branch from b9aae34 to 6ab8de4 Compare June 30, 2022 02:42

prodanlabs requested review from Tingtal and RainbowMango June 30, 2022 02:44

add: disable-scheduler-estimator-in-pull-mode flag

2f35a48

Signed-off-by: prodan <pengshihaoren@gmail.com>

prodanlabs force-pushed the fix-scheduler branch from 6ab8de4 to 2f35a48 Compare June 30, 2022 04:10

karmada-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 30, 2022

karmada-bot assigned RainbowMango Jun 30, 2022

karmada-bot added the lgtm Indicates that a PR is ready to be merged. label Jun 30, 2022

karmada-bot merged commit dc8177c into karmada-io:master Jun 30, 2022

prodanlabs deleted the fix-scheduler branch June 30, 2022 08:54

RainbowMango added this to the v1.3 milestone Aug 30, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

karmada-scheduler add disable-scheduler-estimator-in-pull-mode flag #2064

karmada-scheduler add disable-scheduler-estimator-in-pull-mode flag #2064

prodanlabs commented Jun 25, 2022 •

edited by RainbowMango

Garrybest commented Jun 25, 2022 •

edited

prodanlabs commented Jun 25, 2022 •

edited

Garrybest commented Jun 26, 2022

prodanlabs commented Jun 26, 2022

Garrybest commented Jun 27, 2022

prodanlabs commented Jun 27, 2022

prodanlabs commented Jun 27, 2022

prodanlabs commented Jun 27, 2022

Garrybest commented Jun 27, 2022

prodanlabs commented Jun 27, 2022

prodanlabs commented Jun 27, 2022

Garrybest commented Jun 29, 2022

RainbowMango left a comment

RainbowMango Jun 29, 2022

Tingtal Jun 30, 2022

Tingtal left a comment

Tingtal Jun 30, 2022

Tingtal Jun 30, 2022

prodanlabs Jun 30, 2022

prodanlabs commented Jun 30, 2022

XiShanYongYe-Chang commented Jun 30, 2022

prodanlabs commented Jun 30, 2022

XiShanYongYe-Chang commented Jun 30, 2022

prodanlabs commented Jun 30, 2022

XiShanYongYe-Chang commented Jun 30, 2022

prodanlabs commented Jun 30, 2022

RainbowMango commented Jun 30, 2022

RainbowMango commented Jun 30, 2022

karmada-bot commented Jun 30, 2022

	fs.BoolVar(&o.DisableSchedulerEstimatorInPullMode, "disable-scheduler-estimator-in-pull-mode", false, "Disable scheduler estimator in pull mode clusters, only takes effect when enable-scheduler-estimator is true.")
	fs.BoolVar(&o.DisableSchedulerEstimatorInPullMode, "disable-scheduler-estimator-in-pull-mode", false, "Disable scheduler estimator for pull mode clusters, only takes effect when enable-scheduler-estimator is true.")

	// disableSchedulerEstimatorInPullMode represents whether to disable the pull mode scheduler estimator.
	// disableSchedulerEstimatorInPullMode represents whether to disable the scheduler estimator in pull mode.

karmada-scheduler add disable-scheduler-estimator-in-pull-mode flag #2064

karmada-scheduler add disable-scheduler-estimator-in-pull-mode flag #2064

Conversation

prodanlabs commented Jun 25, 2022 • edited by RainbowMango

Garrybest commented Jun 25, 2022 • edited

prodanlabs commented Jun 25, 2022 • edited

Garrybest commented Jun 26, 2022

prodanlabs commented Jun 26, 2022

Garrybest commented Jun 27, 2022

prodanlabs commented Jun 27, 2022

prodanlabs commented Jun 27, 2022

prodanlabs commented Jun 27, 2022

Garrybest commented Jun 27, 2022

prodanlabs commented Jun 27, 2022

prodanlabs commented Jun 27, 2022

Garrybest commented Jun 29, 2022

RainbowMango left a comment

Choose a reason for hiding this comment

RainbowMango Jun 29, 2022

Choose a reason for hiding this comment

Tingtal Jun 30, 2022

Choose a reason for hiding this comment

Tingtal left a comment

Choose a reason for hiding this comment

Tingtal Jun 30, 2022

Choose a reason for hiding this comment

Tingtal Jun 30, 2022

Choose a reason for hiding this comment

prodanlabs Jun 30, 2022

Choose a reason for hiding this comment

prodanlabs commented Jun 30, 2022

XiShanYongYe-Chang commented Jun 30, 2022

prodanlabs commented Jun 30, 2022

XiShanYongYe-Chang commented Jun 30, 2022

prodanlabs commented Jun 30, 2022

XiShanYongYe-Chang commented Jun 30, 2022

prodanlabs commented Jun 30, 2022

RainbowMango commented Jun 30, 2022

RainbowMango commented Jun 30, 2022

karmada-bot commented Jun 30, 2022

prodanlabs commented Jun 25, 2022 •

edited by RainbowMango

Garrybest commented Jun 25, 2022 •

edited

prodanlabs commented Jun 25, 2022 •

edited