[Bug] [DataQuality] Dataquality Spark Get Datasource Password Error When Password Has $

[qingwli]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

When Datasource has a password has $, running the Data quality job will fail, Data quality Spark job will get the wrong password and cannot connect.

What you expected to happen

Running as expected.

How to reproduce

When I have a Datasource like the below:

"user": "ds",
"password": "d$dfdf$sadfasdf",
"address": "jdbc:mysql://test.com:3306",
"database": "ds",
"jdbcUrl": "jdbc:mysql://test.com:3306/ds",
"driverClassName": "com.mysql.cj.jdbc.Driver",
"validationQuery": "select 1",
"other": "enabledTLSProtocols=TLSv1.2&useSSL=true&",
"props": {
"enabledTLSProtocols": "TLSv1.2",
"useSSL": "true"
}

The password is d$dfdf$sadfasdf

And run data quality job command will like the blow:

root@dolphinscheduler-worker-0:/opt/dolphinscheduler# ${SPARK_HOME2}/bin/spark-submit --master local --driver-cores 1 --driver-memory 512M --num-executors 2 --executor-cores 2 --executor-memory 2G --conf spark.yarn.maxAppAttempts=1 --class org.apache.dolphinscheduler.data.quality.DataQualityApplication --jars /opt/dolphinscheduler/libs/mysql-connector-java-8.0.16.jar,/opt/dolphinscheduler/libs/hadoop-aws-2.9.2.jar,/opt/dolphinscheduler/libs/aws-core-2.17.232.jar,/opt/dolphinscheduler/libs/aws-java-sdk-s3-1.12.160.jar,/opt/dolphinscheduler/libs/aws-java-sdk-core-1.12.160.jar,/opt/dolphinscheduler/libs/kyuubi-hive-jdbc-1.6.0-incubating.jar,/opt/dolphinscheduler/libs/libthrift-0.9.3.jar /opt/dolphinscheduler/libs/dolphinscheduler-data-quality-3.1.2-SNAPSHOT.jar "{"name":"$t(table_count_check)","env":{"type":"batch","config":null},"readers":[{"type":"JDBC","config":{"database":"ds","password":"d$dfdf$sadfasdf","driver":"com.mysql.cj.jdbc.Driver","user":"ds","output_table":"ds_t_ds_version","table":"t_ds_version","url":"jdbc:mysql://test.com:3306/ds?enabledTLSProtocols=TLSv1.2&useSSL=true&&allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false"} },{"type":"JDBC","config":{"database":"auth","password":"d$dfdf$sadfasdf","driver":"com.mysql.cj.jdbc.Driver","user":"ds","output_table":"ds_data_related_policy","table":"data_related_policy","url":"jdbc:mysql://test.com:3306/ds?enabledTLSProtocols=TLSv1.2&useSSL=true&&allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false"} }],"transformers":[{"type":"sql","config":{"index":1,"output_table":"table_count","sql":"SELECT COUNT(*) AS total FROM ds_t_ds_version "} }],"writers":[{"type":"JDBC","config":{"database":"ds","password":"d$dfdf$sadfasdf","driver":"com.mysql.cj.jdbc.Driver","user":"ds","table":"t_ds_dq_execute_result","url":"jdbc:mysql://test.com:3306/ds?characterEncoding=utf8&useSSL=true&allowMultiQueries=true&rewriteBatchedStatements=true&autoreconnect=true&enabledTLSProtocols=TLSv1.2&allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false","sql":"select 0 as rule_type,'$t(table_count_check)' as rule_name,0 as process_definition_id,233064 as process_instance_id,270001 as task_instance_id,table_count.total AS statistics_value,2 AS comparison_value,1 AS comparison_type,0 as check_type,2 as threshold,0 as operator,0 as failure_strategy,'hdfs://mycluster:8020/user/dolphinscheduler/data_quality_error_data/0_233064_test' as error_output_path,'2023-02-27 07:49:32' as create_time,'2023-02-27 07:49:32' as update_time from table_count "} },{"type":"JDBC","config":{"database":"ds","password":"d$dfdf$sadfasdf","driver":"com.mysql.cj.jdbc.Driver","user":"ds","table":"t_ds_dq_task_statistics_value","url":"jdbc:mysql://test.com:3306/ds?characterEncoding=utf8&useSSL=true&allowMultiQueries=true&rewriteBatchedStatements=true&autoreconnect=true&enabledTLSProtocols=TLSv1.2&allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false","sql":"select 0 as process_definition_id,270001 as task_instance_id,10 as rule_id,'EOHXEHAM25BPOQ5WD+AFF52I3CT78L3VCC6V97Y1N44=' as unique_code,'table_count.total'AS statistics_name,table_count.total AS statistics_value,'2023-02-27 07:49:32' as data_time,'2023-02-27 07:49:32' as create_time,'2023-02-27 07:49:32' as update_time from table_count"} }]}"

And I print the arg[0] in here

dolphinscheduler/dolphinscheduler-data-quality/src/main/java/org/apache/dolphinscheduler/data/quality/DataQualityApplication.java

Lines 42 to 49 in 9668c84

	public static void main(String[] args) throws Exception {
	if (args.length < 1) {
	log.error("Can not find DataQualityConfiguration");
	System.exit(-1);
	}
	String dataQualityParameter = args[0];

I will get the arg:

{"name":"(table_count_check)","env":{"type":"batch","config":null},"readers":[{"type":"JDBC","config":{"database":"ds","password":"d","driver":"com.mysql.cj.jdbc.Driver","user":"ds","output_table":"ds_t_ds_version","table":"t_ds_version","url":"jdbc:mysql://test.com:3306/ds?enabledTLSProtocols=TLSv1.2&useSSL=true&&allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false"} },{"type":"JDBC","config":{"database":"auth","password":"d","driver":"com.mysql.cj.jdbc.Driver","user":"ds","output_table":"ds_data_related_policy","table":"data_related_policy","url":"jdbc:mysql://test.com:3306/ds?enabledTLSProtocols=TLSv1.2&useSSL=true&&allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false"} }],"transformers":[{"type":"sql","config":{"index":1,"output_table":"table_count","sql":"SELECT COUNT(*) AS total FROM ds_t_ds_version "} }],"writers":[{"type":"JDBC","config":{"database":"ds","password":"d","driver":"com.mysql.cj.jdbc.Driver","user":"ds","table":"t_ds_dq_execute_result","url":"jdbc:mysql://test.com:3306/ds?characterEncoding=utf8&useSSL=true&allowMultiQueries=true&rewriteBatchedStatements=true&autoreconnect=true&enabledTLSProtocols=TLSv1.2&allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false","sql":"select 0 as rule_type,'(table_count_check)' as rule_name,0 as process_definition_id,233064 as process_instance_id,270001 as task_instance_id,table_count.total AS statistics_value,2 AS comparison_value,1 AS comparison_type,0 as check_type,2 as threshold,0 as operator,0 as failure_strategy,'hdfs://mycluster:8020/user/dolphinscheduler/data_quality_error_data/0_233064_test' as error_output_path,'2023-02-27 07:49:32' as create_time,'2023-02-27 07:49:32' as update_time from table_count "} },{"type":"JDBC","config":{"database":"ds","password":"d","driver":"com.mysql.cj.jdbc.Driver","user":"ds","table":"t_ds_dq_task_statistics_value","url":"jdbc:mysql://test.com:3306/ds?characterEncoding=utf8&useSSL=true&allowMultiQueries=true&rewriteBatchedStatements=true&autoreconnect=true&enabledTLSProtocols=TLSv1.2&allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false","sql":"select 0 as process_definition_id,270001 as task_instance_id,10 as rule_id,'EOHXEHAM25BPOQ5WD+AFF52I3CT78L3VCC6V97Y1N44=' as unique_code,'table_count.total'AS statistics_name,table_count.total AS statistics_value,'2023-02-27 07:49:32' as data_time,'2023-02-27 07:49:32' as create_time,'2023-02-27 07:49:32' as update_time from table_count"} }]}

And the password change to d

Anything else

It looks like character $ will be escaped by the system, I will fix this bug.

Version

dev

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Thank you for your feedback, we have received your issue, Please wait patiently for a reply.

• In order for us to understand your request as soon as possible, please provide detailed information, version or pictures.
• If you haven't received a reply for a long time, you can join our slack and send your question to channel #troubleshooting