DataX 通道能力更新（MaxCompute、Hologres、Tdengine、OSS）、安全漏洞更新、通用打包更新等

NOTICE

@@ -0,0 +1,39 @@
+========================================================
+DataX 是阿里云 DataWorks数据集成 的开源版本，在阿里巴巴集团内被广泛使用的离线数据同步工具/平台。DataX 实现了包括 MySQL、Oracle、OceanBase、SqlServer、Postgre、HDFS、Hive、ADS、HBase、TableStore(OTS)、MaxCompute(ODPS)、Hologres、DRDS 等各种异构数据源之间高效的数据同步功能。
+
+DataX is an open source offline data synchronization tool / platform widely used in Alibaba group and other companies. DataX implements efficient data synchronization between heterogeneous data sources including mysql, Oracle, oceanbase, sqlserver, postgre, HDFS, hive, ads, HBase, tablestore (OTS), maxcompute (ODPs), hologres, DRDS, etc.
+
+Copyright 1999-2022 Alibaba Group Holding Ltd.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+===================================================================
+文级别引用，按许可证
+This product contains various third-party components under other open source licenses.
+This section summarizes those components and their licenses. 
+GNU Lesser General Public License
+--------------------------------------
+opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/conn/CliQuery.java
+opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/conn/Connection4TSDB.java
+opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/conn/DataPoint4TSDB.java
+opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/conn/DumpSeries.java
+opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/conn/OpenTSDBConnection.java
+opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/conn/OpenTSDBDump.java
+opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/opentsdbreader/Constant.java
+opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/opentsdbreader/Key.java
+opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/opentsdbreader/OpenTSDBReader.java
+opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/opentsdbreader/OpenTSDBReaderErrorCode.java
+opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/util/HttpUtils.java
+opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/util/TSDBUtils.java
+opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/util/TimeUtils.java
+===================================================================

README.md

@@ -3,7 +3,7 @@
 
 # DataX
 
-DataX 是阿里云 [DataWorks数据集成](https://www.aliyun.com/product/bigdata/ide) 的开源版本，在阿里巴巴集团内被广泛使用的离线数据同步工具/平台。DataX 实现了包括 MySQL、Oracle、SqlServer、Postgre、HDFS、Hive、ADS、HBase、TableStore(OTS)、MaxCompute(ODPS)、Hologres、DRDS 等各种异构数据源之间高效的数据同步功能。
+DataX 是阿里云 [DataWorks数据集成](https://www.aliyun.com/product/bigdata/ide) 的开源版本，在阿里巴巴集团内被广泛使用的离线数据同步工具/平台。DataX 实现了包括 MySQL、Oracle、OceanBase、SqlServer、Postgre、HDFS、Hive、ADS、HBase、TableStore(OTS)、MaxCompute(ODPS)、Hologres、DRDS 等各种异构数据源之间高效的数据同步功能。
 
 # DataX 商业版本
 阿里云DataWorks数据集成是DataX团队在阿里云上的商业化产品，致力于提供复杂网络环境下、丰富的异构数据源之间高速稳定的数据移动能力，以及繁杂业务背景下的数据同步解决方案。目前已经支持云上近3000家客户，单日同步数据超过3万亿条。DataWorks数据集成目前支持离线50+种数据源，可以进行整库迁移、批量上云、增量同步、分库分表等各类同步解决方案。2020年更新实时同步能力，2020年更新实时同步能力，支持10+种数据源的读写任意组合。提供MySQL，Oracle等多种数据源到阿里云MaxCompute，Hologres等大数据引擎的一键全增量同步解决方案。
@@ -39,14 +39,15 @@ DataX目前已经有了比较全面的插件体系，主流的RDBMS数据库、N
 | ------------ | ---------- | :-------: | :-------: |:-------: |
 | RDBMS 关系型数据库 | MySQL      |     √     |     √     |[读](https://github.com/alibaba/DataX/blob/master/mysqlreader/doc/mysqlreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/mysqlwriter/doc/mysqlwriter.md)|
 |              | Oracle     |     √     |     √     |[读](https://github.com/alibaba/DataX/blob/master/oraclereader/doc/oraclereader.md) 、[写](https://github.com/alibaba/DataX/blob/master/oraclewriter/doc/oraclewriter.md)|
+|              | OceanBase  |     √     |     √     |[读](https://open.oceanbase.com/docs/community/oceanbase-database/V3.1.0/use-datax-to-full-migration-data-to-oceanbase) 、[写](https://open.oceanbase.com/docs/community/oceanbase-database/V3.1.0/use-datax-to-full-migration-data-to-oceanbase)|
 |              | SQLServer  |     √     |     √     |[读](https://github.com/alibaba/DataX/blob/master/sqlserverreader/doc/sqlserverreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/sqlserverwriter/doc/sqlserverwriter.md)|
 |              | PostgreSQL |     √     |     √     |[读](https://github.com/alibaba/DataX/blob/master/postgresqlreader/doc/postgresqlreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/postgresqlwriter/doc/postgresqlwriter.md)|
 |              | DRDS |     √     |     √     |[读](https://github.com/alibaba/DataX/blob/master/drdsreader/doc/drdsreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/drdswriter/doc/drdswriter.md)|
 |              | 通用RDBMS(支持所有关系型数据库)         |     √     |     √     |[读](https://github.com/alibaba/DataX/blob/master/rdbmsreader/doc/rdbmsreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/rdbmswriter/doc/rdbmswriter.md)|
 | 阿里云数仓数据存储    | ODPS       |     √     |     √     |[读](https://github.com/alibaba/DataX/blob/master/odpsreader/doc/odpsreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/odpswriter/doc/odpswriter.md)|
 |              | ADS        |           |     √     |[写](https://github.com/alibaba/DataX/blob/master/adswriter/doc/adswriter.md)|
 |              | OSS        |     √     |     √     |[读](https://github.com/alibaba/DataX/blob/master/ossreader/doc/ossreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/osswriter/doc/osswriter.md)|
-|              | OCS        |     √     |     √     |[读](https://github.com/alibaba/DataX/blob/master/ocsreader/doc/ocsreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/ocswriter/doc/ocswriter.md)|
+|              | OCS        |           |     √     |[写](https://github.com/alibaba/DataX/blob/master/ocswriter/doc/ocswriter.md)|
 | NoSQL数据存储    | OTS        |     √     |     √     |[读](https://github.com/alibaba/DataX/blob/master/otsreader/doc/otsreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/otswriter/doc/otswriter.md)|
 |              | Hbase0.94  |     √     |     √     |[读](https://github.com/alibaba/DataX/blob/master/hbase094xreader/doc/hbase094xreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/hbase094xwriter/doc/hbase094xwriter.md)|
 |              | Hbase1.1   |     √     |     √     |[读](https://github.com/alibaba/DataX/blob/master/hbase11xreader/doc/hbase11xreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/hbase11xwriter/doc/hbase11xwriter.md)|
@@ -61,6 +62,7 @@ DataX目前已经有了比较全面的插件体系，主流的RDBMS数据库、N
 |              | Elasticsearch       |         |     √     |[写](https://github.com/alibaba/DataX/blob/master/elasticsearchwriter/doc/elasticsearchwriter.md)|
 | 时间序列数据库 | OpenTSDB | √ |  |[读](https://github.com/alibaba/DataX/blob/master/opentsdbreader/doc/opentsdbreader.md)|
 |  | TSDB | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/tsdbreader/doc/tsdbreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/tsdbwriter/doc/tsdbhttpwriter.md)|
+|  | TDengine | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/tdenginereader/doc/tdenginereader-CN.md) 、[写](https://github.com/alibaba/DataX/blob/master/tdenginewriter/doc/tdenginewriter-CN.md)|
 
 # 阿里云DataWorks数据集成
 

adbpgwriter/src/main/doc/adbpgwriter.md

@@ -65,9 +65,9 @@ COPY命令将数据写入ADB PG数据库中。
             "writer": {
               "name": "adbpgwriter",
               "parameter": {
-                  "username": "username",
-                  "password": "password",
-                  "host": "host",
+                  "username": "",
+                  "password": "",
+                  "host": "127.0.0.1",
                   "port": "1234",
                   "database": "database",
                   "schema": "schema",

common/pom.xml

@@ -61,6 +61,14 @@
     </dependencies>
 
     <build>
+        <resources>
+          <resource>
+            <directory>src/main/java</directory>
+            <includes>
+              <include>**/*.properties</include>
+            </includes>
+          </resource>
+        </resources>
         <plugins>
             <plugin>
                 <artifactId>maven-compiler-plugin</artifactId>

common/src/main/java/com/alibaba/datax/common/element/BoolColumn.java

@@ -92,7 +92,13 @@ public Date asDate() {
 		throw DataXException.asDataXException(
 				CommonErrorCode.CONVERT_NOT_SUPPORT, "Bool类型不能转为Date .");
 	}
-
+
+	@Override
+	public Date asDate(String dateFormat) {
+		throw DataXException.asDataXException(
+				CommonErrorCode.CONVERT_NOT_SUPPORT, "Bool类型不能转为Date .");
+	}
+
 	@Override
 	public byte[] asBytes() {
 		throw DataXException.asDataXException(

common/src/main/java/com/alibaba/datax/common/element/BytesColumn.java

@@ -75,6 +75,12 @@ public Date asDate() {
 		throw DataXException.asDataXException(
 				CommonErrorCode.CONVERT_NOT_SUPPORT, "Bytes类型不能转为Date .");
 	}
+
+	@Override
+	public Date asDate(String dateFormat) {
+		throw DataXException.asDataXException(
+				CommonErrorCode.CONVERT_NOT_SUPPORT, "Bytes类型不能转为Date .");
+	}
 
 	@Override
 	public Boolean asBoolean() {

common/src/main/java/com/alibaba/datax/common/element/Column.java

@@ -55,6 +55,8 @@ protected void setByteSize(int byteSize) {
 	public abstract String asString();
 
 	public abstract Date asDate();
+
+	public abstract Date asDate(String dateFormat);
 
 	public abstract byte[] asBytes();
 

common/src/main/java/com/alibaba/datax/common/element/ColumnCast.java

@@ -22,6 +22,11 @@ public static Date string2Date(final StringColumn column)
 			throws ParseException {
 		return StringCast.asDate(column);
 	}
+
+	public static Date string2Date(final StringColumn column, String dateFormat)
+			throws ParseException {
+		return StringCast.asDate(column, dateFormat);
+	}
 
 	public static byte[] string2Bytes(final StringColumn column)
 			throws UnsupportedEncodingException {
@@ -115,6 +120,16 @@ static Date asDate(final StringColumn column) throws ParseException {
 		}
 		throw e;
 	}
+
+	static Date asDate(final StringColumn column, String dateFormat) throws ParseException {
+		ParseException e;
+		try {
+			return FastDateFormat.getInstance(dateFormat, StringCast.timeZoner).parse(column.asString());
+		} catch (ParseException ignored) {
+			e = ignored;
+		}
+		throw e;
+	}
 
 	static byte[] asBytes(final StringColumn column)
 			throws UnsupportedEncodingException {

common/src/main/java/com/alibaba/datax/common/element/DateColumn.java

@@ -89,6 +89,11 @@ public Date asDate() {
 
 		return new Date((Long)this.getRawData());
 	}
+
+	@Override
+	public Date asDate(String dateFormat) {
+		return asDate();
+	}
 
 	@Override
 	public byte[] asBytes() {

common/src/main/java/com/alibaba/datax/common/element/DoubleColumn.java

@@ -132,6 +132,12 @@ public Date asDate() {
 		throw DataXException.asDataXException(
 				CommonErrorCode.CONVERT_NOT_SUPPORT, "Double类型无法转为Date类型 .");
 	}
+
+	@Override
+	public Date asDate(String dateFormat) {
+		throw DataXException.asDataXException(
+				CommonErrorCode.CONVERT_NOT_SUPPORT, "Double类型无法转为Date类型 .");
+	}
 
 	@Override
 	public byte[] asBytes() {

common/src/main/java/com/alibaba/datax/common/element/LongColumn.java

@@ -125,6 +125,11 @@ public Date asDate() {
 		}
 		return new Date(this.asLong());
 	}
+
+	@Override
+	public Date asDate(String dateFormat) {
+		return this.asDate();
+	}
 
 	@Override
 	public byte[] asBytes() {

common/src/main/java/com/alibaba/datax/common/element/Record.java

@@ -1,5 +1,7 @@
 package com.alibaba.datax.common.element;
 
+import java.util.Map;
+
 /**
  * Created by jingxing on 14-8-24.
  */
@@ -20,4 +22,8 @@ public interface Record {
 
 	public int getMemorySize();
 
+	public void setMeta(Map<String, String> meta);
+
+	public Map<String, String> getMeta();
+
 }

common/src/main/java/com/alibaba/datax/common/element/StringColumn.java

@@ -149,6 +149,16 @@ public Date asDate() {
 					String.format("String[\"%s\"]不能转为Date .", this.asString()));
 		}
 	}
+
+	@Override
+	public Date asDate(String dateFormat) {
+		try {
+			return ColumnCast.string2Date(this, dateFormat);
+		} catch (Exception e) {
+			throw DataXException.asDataXException(CommonErrorCode.CONVERT_NOT_SUPPORT,
+					String.format("String[\"%s\"]不能转为Date .", this.asString()));
+		}
+	}
 
 	@Override
 	public byte[] asBytes() {

common/src/main/java/com/alibaba/datax/common/exception/DataXException.java

@@ -16,6 +16,10 @@ public DataXException(ErrorCode errorCode, String errorMessage) {
         this.errorCode = errorCode;
     }
 
+    public DataXException(String errorMessage) {
+        super(errorMessage);
+    }
+
     private DataXException(ErrorCode errorCode, String errorMessage, Throwable cause) {
         super(errorCode.toString() + " - " + getMessage(errorMessage) + " - " + getMessage(cause), cause);
 
@@ -26,6 +30,10 @@ public static DataXException asDataXException(ErrorCode errorCode, String messag
         return new DataXException(errorCode, message);
     }
 
+    public static DataXException asDataXException(String message) {
+        return new DataXException(message);
+    }
+
     public static DataXException asDataXException(ErrorCode errorCode, String message, Throwable cause) {
         if (cause instanceof DataXException) {
             return (DataXException) cause;

common/src/main/java/com/alibaba/datax/common/plugin/AbstractPlugin.java

@@ -3,6 +3,8 @@
 import com.alibaba.datax.common.base.BaseObject;
 import com.alibaba.datax.common.util.Configuration;
 
+import java.util.List;
+
 public abstract class AbstractPlugin extends BaseObject implements Pluginable {
 	//作业的config
     private Configuration pluginJobConf;
@@ -15,6 +17,8 @@ public abstract class AbstractPlugin extends BaseObject implements Pluginable {
 
     private String peerPluginName;
 
+    private List<Configuration> readerPluginSplitConf;
+
     @Override
 	public String getPluginName() {
 		assert null != this.pluginConf;
@@ -84,4 +88,12 @@ public void preHandler(Configuration jobConfiguration){
     public void postHandler(Configuration jobConfiguration){
 
     }
+
+    public List<Configuration> getReaderPluginSplitConf(){
+        return this.readerPluginSplitConf;
+    }
+
+    public void setReaderPluginSplitConf(List<Configuration> readerPluginSplitConf){
+        this.readerPluginSplitConf = readerPluginSplitConf;
+    }
 }

common/src/main/java/com/alibaba/datax/common/util/ConfigurationUtil.java

@@ -0,0 +1,37 @@
+package com.alibaba.datax.common.util;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.commons.lang3.StringUtils;
+
+public class ConfigurationUtil {
+    private static final List<String> SENSITIVE_KEYS = Arrays.asList("password", "accessKey", "securityToken",
+        "AccessKeyId", "AccessKeySecert", "AccessKeySecret", "clientPassword");
+
+    public static Configuration filterSensitive(Configuration origin) {
+        // shell 任务configuration metric 可能为null。
+        if (origin == null) {
+            return origin;
+        }
+        // 确保不影响入参的对象
+        Configuration configuration = origin.clone();
+        Set<String> keys = configuration.getKeys();
+        for (final String key : keys) {
+            boolean isSensitive = false;
+            for (String sensitiveKey : SENSITIVE_KEYS) {
+                if (StringUtils.endsWithIgnoreCase(key, sensitiveKey)) {
+                    isSensitive = true;
+                    break;
+                }
+            }
+
+            if (isSensitive && configuration.get(key) instanceof String) {
+                configuration.set(key, configuration.getString(key).replaceAll(".", "*"));
+            }
+
+        }
+        return configuration;
+    }
+}