[Bug] [Task Plugin] AliyunServerlessSpark datasource plugin uses outdated EMR SDK (1.0.0), causing NoSuchMethodError

[includetts]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

The dolphinscheduler-datasource-aliyunserverlessspark plugin declares emr_serverless_spark20230808:1.0.0, while dolphinscheduler-task-aliyunserverlessspark declares version 2.4.1.

At runtime, both plugins are loaded as shade jars. Due to classpath ordering (datasource-plugins/ directory comes before task-plugins/), the JVM loads com.aliyun.emr_serverless_spark20230808.Client from the datasource shade jar (version 1.0.0). This version does not contain the getTemplate() method that the task plugin code calls, resulting in NoSuchMethodError at runtime.

What you expected to happen

The datasource plugin should use the same emr_serverless_spark20230808 SDK version (2.4.1) as the task plugin, so no class loading conflicts occur.

Error log:
java.lang.NoSuchMethodError: com.aliyun.emr_serverless_spark20230808.Client.getTemplate(Ljava/lang/String;Lcom/aliyun/emr_serverless_spark20230808/models/GetTemplateRequest;)Lcom/aliyun/emr_serverless_spark20230808/models/GetTemplateResponse;

How to reproduce

1. Deploy DolphinScheduler (standalone or cluster mode)
2. 2. Install both dolphinscheduler-datasource-aliyunserverlessspark and dolphinscheduler-task-aliyunserverlessspark plugins
3. 3. Create an AliyunServerlessSpark datasource connection
4. 4. Submit an AliyunServerlessSpark task
5. 5. The task fails with NoSuchMethodError: Client.getTemplate()
      Root cause is in dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-aliyunserverlessspark/pom.xml:

• Current: emr_serverless_spark20230808 version 1.0.0
• • Should be: emr_serverless_spark20230808 version 2.4.1 (same as task plugin)

Anything else

This bug occurs every time when using the AliyunServerlessSpark task plugin. The fix is straightforward: upgrade the EMR SDK version in the datasource plugin pom.xml from 1.0.0 to 2.4.1.

Version

dev

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct