[Feature][Deployment] Add KEDA autoscaler support for worker deployment when deployed in K8S cluster #13367
Conversation
Codecov Report
@@ Coverage Diff @@
## dev #13367 +/- ##
============================================
- Coverage 39.45% 39.43% -0.02%
+ Complexity 4313 4311 -2
============================================
Files 1083 1083
Lines 40756 40778 +22
Branches 4670 4674 +4
============================================
+ Hits 16079 16081 +2
- Misses 22890 22910 +20
Partials 1787 1787
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
deploy/kubernetes/dolphinscheduler/templates/keda-autoscaler-worker.yaml
Outdated
Show resolved
Hide resolved
|
Action Items for this PR:
|
EricGao888
requested review from
zhongjiajie,
Tianqi-Dotes and
caishunfeng
as code owners
| sslmode: "disable" | ||
| targetQueryValue: "1" | ||
| query: >- | ||
| SELECT ceil(COUNT(*)::decimal / {{ .Values.worker.env.WORKER_EXEC_THREADS }}) |
There was a problem hiding this comment.
There is a tricky issue that users can change the worker exec threads through a configmap, yet I don't come out a solution for that case 😅
There was a problem hiding this comment.
Good catch. I don't have a solution at this moment either. I will add a reminder in docs for users about this.
| dbName: {{ .Values.mysql.auth.database }} | ||
| username: {{ .Values.mysql.auth.username }} | ||
| passwordFromEnv: SPRING_DATASOURCE_PASSWORD | ||
| queryValue: "1" |
There was a problem hiding this comment.
This key queryValue is different from the one in postgresql targetQueryValue, is that expected?
There was a problem hiding this comment.
This key
queryValueis different from the one in postgresqltargetQueryValue, is that expected?
Yes, it is. As you could see in the following source code, postgresql and mysql autoscalers have different names for the same parameter.
| {{- else if .Values.externalDatabase.enabled }} | ||
| - type: mysql |
There was a problem hiding this comment.
An externalDatabase is not necessarily mysql 😅 it can be postgresql as well, you might want to use
| {{- else if .Values.externalDatabase.enabled }} | |
| - type: mysql | |
| {{- else if .Values.externalDatabase.enabled }} | |
| - type: {{ .Values.externalDatabase.type }} |
There was a problem hiding this comment.
An
externalDatabaseis not necessarilymysql😅 it can be postgresql as well, you might want to use
I will use a condition clause here to support both postgresql and mysql. As these two use different autoscalers and there are tiny differences.
| which will significantly save the resources. | ||
|
|
||
| Worker autoscaling feature is compatible with `postgresql` and `mysql` shipped with `DolphinScheduler official helm chart`. If you | ||
| use external database, worker autoscaling feature only supports external `mysql` databases. |
There was a problem hiding this comment.
Do we have trouble to support both mysql and postgresql now?
There was a problem hiding this comment.
No trouble. Will add support for postgresql in next commit.
|
Kudos, SonarCloud Quality Gate passed!
|
Purpose of the pull request
KEDA.Brief change log
Verify this pull request