3.2.2 release cherry-pick cp: Reduce the size of tarball to continue ASF release #16042

wangxj3 · 2024-05-21T02:35:10Z

Purpose of the pull request

It is prepared for version 3.2.2, mainly to reduce the size of the release package and fix the helm version problem.

Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(or)

If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

* Fix rpc channel leak due to concurrent operation * Throw channel create failed exception --------- Co-authored-by: Rick Cheng <rickchengx@gmail.com>

…apache#15540) * Reduce the size of tarball to continue ASF release for more detail you can see https://lists.apache.org/thread/rmp7fghlj0n7h9y2v3p8gkw9f9qbo6qt # Conflicts: # tools/dependencies/known-dependencies.txt

…6041)

Co-authored-by: Eric Gao <ericgao.apache@gmail.com>

…ers (apache#16033) Co-authored-by: Rick Cheng <rickchengx@gmail.com>

ruanwenjun · 2024-05-21T07:36:07Z

Build failed

…/project parameters data type (apache#15967) * [Improvement][parameter] New data types and type filtering * [Improvement][parameter] Improvement startup parameters/global parameters data type * fix api interfaces compatible * add project parameter data type default value * [Improvement][parameter] New data types and type filtering * [Improvement][parameter] Improvement startup parameters/global parameters data type * fix api interfaces compatible * add project parameter data type default value * improvement project code * remove useless imports * remove method onClearSearchTaskType * add parameter doc * optimisation logic * code conflict resolution * code conflict resolution * [Improvement][Monitor] Show master && worker Busy Or Normal Status and Show Commands table list (apache#15978) * update * test * add monitor enhance ui * update * update * update doc * fix spotless * update * update * Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/DataAnalysisController.java Co-authored-by: Wenjun Ruan <wenjun@apache.org> * Update dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/DataAnalysisController.java Co-authored-by: Wenjun Ruan <wenjun@apache.org> * Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.java Co-authored-by: Wenjun Ruan <wenjun@apache.org> * Update dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.xml Co-authored-by: Wenjun Ruan <wenjun@apache.org> * Update dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/mapper/CommandMapper.java Co-authored-by: Wenjun Ruan <wenjun@apache.org> * Update dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/ErrorCommandMapper.xml Co-authored-by: Wenjun Ruan <wenjun@apache.org> * update * fix spotless * update --------- Co-authored-by: Wenjun Ruan <wenjun@apache.org> * [Improvement][Monitor] Add UT for montor (apache#15998) * formatting Code

…che#16038)

Gallardot · 2024-05-23T02:01:35Z

deploy/kubernetes/dolphinscheduler/Chart.yaml


 # This is the version number of the application being deployed. This version number should be
 # incremented each time you make changes to the application.
-appVersion: 3.1.0
+appVersion: 3.2.2


I don't think this appVersion should be changed.
dolphinscheduler.master.labels quotes appVersion

dolphinscheduler/deploy/kubernetes/dolphinscheduler/templates/_helpers.tpl

Lines 51 to 64 in 56ecec7

{{- define "dolphinscheduler.common.labels" -}}

app.kubernetes.io/instance: {{ .Release.Name }}

app.kubernetes.io/managed-by: {{ .Release.Service }}

app.kubernetes.io/version: {{ .Chart.AppVersion }}

{{- end -}}

{{/*

Create a master labels.

*/}}

{{- define "dolphinscheduler.master.labels" -}}

app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-master

app.kubernetes.io/component: master

{{ include "dolphinscheduler.common.labels" . }}

{{- end -}}

And dolphinscheduler.master.labels as master's matchLabels, but matchLabels are immutable, which means that, Updating helm charts for an already deployed dolphinscheduler will fail

dolphinscheduler/deploy/kubernetes/dolphinscheduler/templates/statefulset-dolphinscheduler-master.yaml

Lines 27 to 29 in 56ecec7

selector:

matchLabels:

{{- include "dolphinscheduler.master.labels" . | nindent 6 }}

If we still want to modify the appVersion, then we should remove the app.kubernetes.io/version: {{ .Chart.AppVersion }} from dolphinscheduler.common.labels first

I have created a PR #16066 to fix this issue.
After merging, we can modify the appversion as normal.

@wangxj3 Please remember to cherry pick this PR #16066 before 3.2.2 is released

SbloodyS · 2024-05-23T14:58:47Z

dolphinscheduler-api-test/dolphinscheduler-api-test-case/pom.xml

+        <dependency>
+            <groupId>org.apache.httpcomponents</groupId>
+            <artifactId>httpclient</artifactId>
+            <version>4.5.13</version>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.httpcomponents</groupId>
+            <artifactId>httpmime</artifactId>
+            <version>4.5.13</version>
+        </dependency>


I don't see any related test cases being added. Why add this?

SbloodyS · 2024-05-23T14:59:49Z

dolphinscheduler-api/pom.xml

@@ -74,7 +74,7 @@

        <dependency>
            <groupId>org.apache.dolphinscheduler</groupId>
-            <artifactId>dolphinscheduler-task-all</artifactId>
+            <artifactId>dolphinscheduler-task-all-prune</artifactId>


What is this?

reduce release package size

Co-authored-by: xiangzihao <460888207@qq.com>

Fix the PreparedStatement parameter is TIME, set it to java.sql.Time Co-authored-by: Aaron Wang <wangweirao16@gmail.com> Co-authored-by: Rick Cheng <rickchengx@gmail.com>

SbloodyS · 2024-05-24T05:52:12Z

dolphinscheduler-task-plugin/dolphinscheduler-task-all-prune/pom.xml

+            <version>${project.version}</version>
+            <exclusions>
+                <exclusion>
+                    <groupId>*</groupId>
+                    <artifactId>*</artifactId>
+                </exclusion>
+            </exclusions>
+        </dependency>
+
+        <dependency>
+            <groupId>org.apache.dolphinscheduler</groupId>
+            <artifactId>dolphinscheduler-task-hivecli</artifactId>
+            <version>${project.version}</version>
+            <exclusions>
+                <exclusion>
+                    <groupId>*</groupId>
+                    <artifactId>*</artifactId>
+                </exclusion>
+            </exclusions>
+        </dependency>
+
+        <dependency>
+            <groupId>org.apache.dolphinscheduler</groupId>
+            <artifactId>dolphinscheduler-task-chunjun</artifactId>
+            <version>${project.version}</version>
+            <exclusions>
+                <exclusion>
+                    <groupId>*</groupId>
+                    <artifactId>*</artifactId>
+                </exclusion>
+            </exclusions>
+        </dependency>
+
+        <dependency>
+            <groupId>org.apache.dolphinscheduler</groupId>
+            <artifactId>dolphinscheduler-task-dms</artifactId>
+            <version>${project.version}</version>
+            <exclusions>
+                <exclusion>
+                    <groupId>*</groupId>
+                    <artifactId>*</artifactId>
+                </exclusion>
+            </exclusions>
+        </dependency>
+
+        <dependency>
+            <groupId>org.apache.dolphinscheduler</groupId>
+            <artifactId>dolphinscheduler-task-datasync</artifactId>
+            <version>${project.version}</version>
+            <exclusions>
+                <exclusion>
+                    <groupId>*</groupId>
+                    <artifactId>*</artifactId>
+                </exclusion>
+            </exclusions>
+        </dependency>
+
+        <dependency>
+            <groupId>org.apache.dolphinscheduler</groupId>
+            <artifactId>dolphinscheduler-task-kubeflow</artifactId>
+            <version>${project.version}</version>
+            <exclusions>
+                <exclusion>
+                    <groupId>*</groupId>
+                    <artifactId>*</artifactId>
+                </exclusion>
+            </exclusions>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.dolphinscheduler</groupId>
+            <artifactId>dolphinscheduler-task-linkis</artifactId>
+            <version>${project.version}</version>
+            <exclusions>
+                <exclusion>
+                    <groupId>*</groupId>
+                    <artifactId>*</artifactId>
+                </exclusion>
+            </exclusions>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.dolphinscheduler</groupId>
+            <artifactId>dolphinscheduler-task-datafactory</artifactId>
+            <version>${project.version}</version>
+            <exclusions>
+                <exclusion>
+                    <groupId>*</groupId>
+                    <artifactId>*</artifactId>
+                </exclusion>
+            </exclusions>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.dolphinscheduler</groupId>
+            <artifactId>dolphinscheduler-task-remoteshell</artifactId>
+            <version>${project.version}</version>
+            <exclusions>
+                <exclusion>
+                    <groupId>*</groupId>
+                    <artifactId>*</artifactId>
+                </exclusion>
+            </exclusions>


If we do this, please inform the user in the documentation that this is an incompatible change. This may lead to serious production accidents. cc @ruanwenjun @EricGao888 @Radeity

Yes, this is a sad solution from 3.2.1, since no-one want to refactor the release architecture. Right now, the release package size is 1.x g, large than ASF limit. I create #15960 to do this.

codecov-commenter · 2024-05-24T09:14:08Z

Codecov Report

Attention: Patch coverage is 52.27273% with 42 lines in your changes are missing coverage. Please review.

Project coverage is 40.66%. Comparing base (7b7a333) to head (c025eb7).

❗ Current head c025eb7 differs from pull request most recent head 4ded5ad

Please upload reports for the commit 4ded5ad to get more accurate results.

Files	Patch %	Lines
...scheduler/plugin/task/api/utils/PropertyUtils.java	0.00%	14 Missing ⚠️
...duler/extract/base/client/NettyRemotingClient.java	65.21%	6 Missing and 2 partials ⚠️
...cheduler/common/log/remote/S3RemoteLogHandler.java	0.00%	3 Missing ⚠️
...r/plugin/registry/zookeeper/ZookeeperRegistry.java	40.00%	1 Missing and 2 partials ⚠️
...eduler/service/expand/CuringParamsServiceImpl.java	66.66%	2 Missing and 1 partial ⚠️
...cheduler/api/service/impl/ExecutorServiceImpl.java	0.00%	1 Missing and 1 partial ⚠️
...phinscheduler/plugin/task/emr/AbstractEmrTask.java	0.00%	2 Missing ⚠️
...scheduler/plugin/task/sagemaker/SagemakerTask.java	0.00%	2 Missing ⚠️
...inscheduler/api/controller/ExecutorController.java	50.00%	1 Missing ⚠️
...duler/api/service/impl/WorkerGroupServiceImpl.java	0.00%	0 Missing and 1 partial ⚠️
... and 3 more

Additional details and impacted files

@@                 Coverage Diff                 @@
##             3.2.2-prepare   #16042      +/-   ##
===================================================
+ Coverage            40.58%   40.66%   +0.07%     
- Complexity            5216     5225       +9     
===================================================
  Files                 1380     1380              
  Lines                46056    46011      -45     
  Branches              4916     4919       +3     
===================================================
+ Hits                 18693    18710      +17     
+ Misses               25437    25376      -61     
+ Partials              1926     1925       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

sonarcloud · 2024-05-25T02:40:22Z

Please retry analysis of this Pull-Request directly on SonarCloud

wangxj and others added 6 commits May 16, 2024 17:55

add version of 3.2.2

a3bfb76

Merge branch 'apache:dev' into dev

e0bcfb3

Fix rpc channel leak due to concurrent operation (apache#16021)

5463d02

* Fix rpc channel leak due to concurrent operation * Throw channel create failed exception --------- Co-authored-by: Rick Cheng <rickchengx@gmail.com>

Fix WorkerTaskExecutorThreadPool#isOverload is incorrect (apache#16027)

b184364

Increase block-until-connected in ZookeeperRegistryTestCase (apache#1…

e4df01a

…6041)

wangxj3 requested review from EricGao888, caishunfeng, SbloodyS, zhuangchong and ruanwenjun as code owners May 21, 2024 02:35

github-actions bot added backend test document labels May 21, 2024

rickchengx and others added 2 commits May 21, 2024 11:16

[Fix][CI] fix the ci error of Values.datasource.profile (apache#16031)

dbd790d

Co-authored-by: Eric Gao <ericgao.apache@gmail.com>

Correction of Typos in the Chinese Document Appendix for Task Paramet…

cf400b8

…ers (apache#16033) Co-authored-by: Rick Cheng <rickchengx@gmail.com>

sdhzwc and others added 3 commits May 21, 2024 15:39

add pom

fed94c8

Fix NettyRemotingClient might throw IllegalMonitorStateException (apa…

56ecec7

…che#16038)

Gallardot requested changes May 23, 2024

View reviewed changes

Removed unused StateEventHandleFailure (apache#16052)

6862fe7

SbloodyS reviewed May 23, 2024

View reviewed changes

ruanwenjun and others added 5 commits May 24, 2024 10:34

Remove OmitStackTraceInFastThrow in start.sh (apache#16054)

5037f0a

Update Chart.yaml Rollback APPversion

7b0fa1a

modify log msg (apache#16062)

581a7ed

Co-authored-by: xiangzihao <460888207@qq.com>

[DS-16046][fix] Set PreparedStatement parameter type (apache#16050)

baa252a

Fix the PreparedStatement parameter is TIME, set it to java.sql.Time Co-authored-by: Aaron Wang <wangweirao16@gmail.com> Co-authored-by: Rick Cheng <rickchengx@gmail.com>

Update pom.xml

b83141f

github-actions bot removed the test label May 24, 2024

SbloodyS reviewed May 24, 2024

View reviewed changes

ruanwenjun and others added 2 commits May 24, 2024 14:15

[DSIP-42] Add dolphinscheduler-aws-authentication module (apache#16043)

f754611

Merge branch 'apache:dev' into 3.2.2-release

4ded5ad

wangxj3 requested review from songjianet and Amy0104 as code owners May 24, 2024 08:45

github-actions bot added UI ui and front end related e2e e2e test labels May 24, 2024

songjianet approved these changes May 24, 2024

View reviewed changes

Gallardot mentioned this pull request May 24, 2024

[helm] remove AppVersion from common labels #16066

Merged

[helm] remove appversion from labels (apache#16066)

d5cbf64

wangxj3 merged commit 646bbf4 into apache:3.2.2-prepare May 25, 2024
42 of 44 checks passed

SbloodyS mentioned this pull request May 25, 2024

Revert "3.2.2 release cherry-pick cp: Reduce the size of tarball to continue ASF release" #16068

Merged

wangxj3 deleted the 3.2.2-release branch May 30, 2024 08:27

wangxj3 restored the 3.2.2-release branch May 30, 2024 08:27

wangxj3 deleted the 3.2.2-release branch May 30, 2024 08:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

3.2.2 release cherry-pick cp: Reduce the size of tarball to continue ASF release #16042

3.2.2 release cherry-pick cp: Reduce the size of tarball to continue ASF release #16042

wangxj3 commented May 21, 2024 •

edited

Loading

ruanwenjun commented May 21, 2024

Gallardot May 23, 2024

Gallardot May 23, 2024

Gallardot May 24, 2024

SbloodyS May 23, 2024

wangxj3 May 24, 2024

SbloodyS May 23, 2024

wangxj3 May 24, 2024

SbloodyS May 24, 2024

ruanwenjun May 24, 2024

Radeity May 24, 2024

codecov-commenter commented May 24, 2024

sonarcloud bot commented May 25, 2024

	{{- define "dolphinscheduler.common.labels" -}}
	app.kubernetes.io/instance: {{ .Release.Name }}
	app.kubernetes.io/managed-by: {{ .Release.Service }}
	app.kubernetes.io/version: {{ .Chart.AppVersion }}
	{{- end -}}

	{{/*
	Create a master labels.
	*/}}
	{{- define "dolphinscheduler.master.labels" -}}
	app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-master
	app.kubernetes.io/component: master
	{{ include "dolphinscheduler.common.labels" . }}
	{{- end -}}

	selector:
	matchLabels:
	{{- include "dolphinscheduler.master.labels" . \| nindent 6 }}

3.2.2 release cherry-pick cp: Reduce the size of tarball to continue ASF release #16042

3.2.2 release cherry-pick cp: Reduce the size of tarball to continue ASF release #16042

Conversation

wangxj3 commented May 21, 2024 • edited Loading

Purpose of the pull request

Brief change log

Verify this pull request

ruanwenjun commented May 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented May 24, 2024

Codecov Report

sonarcloud bot commented May 25, 2024

wangxj3 commented May 21, 2024 •

edited

Loading