[enhancement] (odbc) support more ODBC Connection Parameters for odbc external table

docs/en/ecosystem/external-table/odbc-of-doris.md

@@ -110,7 +110,10 @@ Parameter | Description
 **type** | The type of external database, currently supports Oracle, MySQL and PostgerSQL
 **user** | The user name of database
 **password** | password for the user
+**charset** | charset of connection
 
+Remark：
+In addition to adding the above parameters to `PROPERTIES`, you can also add parameters specific to each database's ODBC driver implementation, such as `sslverify` for mysql, etc.
 
 ##### Installation and configuration of ODBC driver
 

docs/en/faq/install-faq.md

@@ -278,3 +278,16 @@ If the same error is reported, Run the following command
 ```
 cp fe-core/target/generated-sources/cup/org/apache/doris/analysis/action_table.dat fe-core/target/classes/org/apache/doris/analysis
 ```
+
+### ### Q14. Doris upgrades to version 1.0 or later and reports error ``Failed to set ciphers to use (2026)` in MySQL appearance via ODBC.
+This problem occurs after doris upgrades to version 1.0 and uses Connector/ODBC 8.0.x or higher. Connector/ODBC 8.0.x has multiple access methods, such as `/usr/lib64/libmyodbc8w.so` which is installed via yum and relies on ` libssl.so.10` and `libcrypto.so.10`.
+In doris 1.0 onwards, openssl has been upgraded to 1.1 and is built into the doris binary package, so this can lead to openssl conflicts and errors like the following
+```
+ERROR 1105 (HY000): errCode = 2, detailMessage = driver connect Error: HY000 [MySQL][ODBC 8.0(w) Driver]SSL connection error: Failed to set ciphers to use (2026)
+```
+The solution is to use the `Connector/ODBC 8.0.28` version of ODBC Connector and select `Linux - Generic` in the operating system, this version of ODBC Driver uses openssl version 1,1. For details, see the [ODBC exterior documentation](. /extending-doris/odbc-of-doris.md)
+You can verify the version of openssl used by MySQL ODBC Driver by
+```
+ldd /path/to/libmyodbc8w.so |grep libssl.so
+```
+If the output contains ``libssl.so.10``, there may be problems using it, if it contains ``libssl.so.1.1``, it is compatible with doris 1.0

docs/en/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-RESOURCE.md

@@ -108,6 +108,8 @@ illustrate:
    - odbc_type: the type of the external database, currently supports oracle, mysql, postgresql
    - user: username of the foreign database
    - password: the password information of the corresponding user
+   - charset: connection charset
+   - There is also support for implementing custom parameters per ODBC Driver, see the description of the corresponding ODBC Driver
 
 3. Create S3 resource
 

docs/zh-CN/ecosystem/external-table/odbc-of-doris.md

@@ -106,6 +106,10 @@ PROPERTIES (
 **odbc_type** | 外表数据库的类型，当前支持oracle, mysql, postgresql
 **user** | 外表数据库的用户名
 **password** | 对应用户的密码信息
+**charset** | 数据库连接使用的字符集
+
+备注：
+`PROPERTIES` 中除了可以添加上述参数之外，还支持每个数据库的ODBC driver 实现的专用参数，比如mysql 的`sslverify` 等
 
 
 

docs/zh-CN/faq/install-faq.md

@@ -277,4 +277,17 @@ cd fe && mvn clean install -DskipTests
 如果还报同样的错误，手动执行如下命令
 ```
 cp fe-core/target/generated-sources/cup/org/apache/doris/analysis/action_table.dat fe-core/target/classes/org/apache/doris/analysis
-```
+```
+
+### Q14. Doris 升级到1.0 以后版本通过ODBC访问MySQL外表报错 `Failed to set ciphers to use (2026)`
+这个问题出现在doris 升级到1.0 版本以后，且使用 Connector/ODBC 8.0.x 以上版本，Connector/ODBC 8.0.x 有多种获取方式，比如通过yum安装的的方式获取的 `/usr/lib64/libmyodbc8w.so` 依赖的是 `libssl.so.10` 和 `libcrypto.so.10`
+而doris 1.0 以后版本中openssl 已经升级到1.1 且内置在doris 二进制包中，因此会导致 openssl 的冲突进而出现 类似 如下的错误
+```
+ERROR 1105 (HY000): errCode = 2, detailMessage = driver connect Error: HY000 [MySQL][ODBC 8.0(w) Driver]SSL connection error: Failed to set ciphers to use (2026)
+```
+解决方式是使用`Connector/ODBC 8.0.28` 版本的 ODBC Connector， 并且选择 在操作系统处选择 `Linux - Generic`, 这个版本的ODBC Driver 使用 openssl 1.1 版本。具体使用方式见 [ODBC外表使用文档](../extending-doris/odbc-of-doris.md)
+可以通过如下方式验证 MySQL ODBC Driver 使用的openssl 版本
+```
+ldd /path/to/libmyodbc8w.so |grep libssl.so
+```
+如果输出包含 `libssl.so.10` 则使用过程中可能出现问题， 如果包含`libssl.so.1.1` 则与doris 1.0 兼容

docs/zh-CN/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-RESOURCE.md

@@ -108,6 +108,8 @@ PROPERTIES ("key"="value", ...);
    - odbc_type：外表数据库的类型，当前支持oracle, mysql, postgresql
    - user：外表数据库的用户名
    - password：对应用户的密码信息
+   - charset: 数据库链接的编码信息
+   - 另外还支持每个ODBC Driver 实现自定义的参数，参见对应ODBC Driver 的说明
 
 3. 创建 S3 resource
 

fe/fe-core/src/main/java/org/apache/doris/catalog/MysqlTable.java

@@ -146,7 +146,7 @@ private String getPropertyFromResource(String propertyName) {
             throw new RuntimeException("Resource does not exist. name: " + odbcCatalogResourceName);
         }
 
-        String property = odbcCatalogResource.getProperties(propertyName);
+        String property = odbcCatalogResource.getProperty(propertyName);
         if (property == null) {
             throw new RuntimeException("The property:" + propertyName + " do not set in resource " + odbcCatalogResourceName);
         }

fe/fe-core/src/main/java/org/apache/doris/catalog/OdbcCatalogResource.java

@@ -91,7 +91,6 @@ private void checkProperties(String propertiesKey) throws DdlException {
         if (value == null) {
             throw new DdlException("Missing " + propertiesKey + " in properties");
         }
-
     }
 
     @Override
@@ -121,7 +120,13 @@ public void checkProperties(Map<String, String> properties) throws AnalysisExcep
         }
     }
 
-    public String getProperties(String propertiesKey)  {
+    @Override
+    public Map<String, String> getCopiedProperties() {
+        Map<String, String> copiedProperties = Maps.newHashMap(configs);
+        return copiedProperties;
+    }
+
+    public String getProperty(String propertiesKey)  {
         // check the properties key
         String value = configs.get(propertiesKey);
         return value;

fe/fe-core/src/main/java/org/apache/doris/catalog/OdbcTable.java

@@ -43,6 +43,8 @@
 import java.util.List;
 import java.util.Map;
 
+import static java.util.stream.Collectors.joining;
+
 public class OdbcTable extends Table {
     private static final Logger LOG = LogManager.getLogger(OlapTable.class);
 
@@ -55,6 +57,9 @@ public class OdbcTable extends Table {
     public static final String ODBC_TABLE = "table";
     public static final String ODBC_DRIVER = "driver";
     public static final String ODBC_TYPE = "odbc_type";
+    public static final String ODBC_CHARSET = "charset";
+    public static final String ODBC_EXTRA_PARAM = "extra_param";
+
 
     // map now odbc external table Doris support now
     private static Map<String, TOdbcTableType> TABLE_TYPE_MAP;
@@ -72,7 +77,7 @@ public class OdbcTable extends Table {
     private static String mysqlProperName(String name) {
         return "`" + name + "`";
     }
-    
+
     public static String databaseProperName(TOdbcTableType tableType, String name) {
         switch (tableType) {
             case MYSQL:
@@ -90,6 +95,9 @@ public static String databaseProperName(TOdbcTableType tableType, String name) {
     private String odbcTableName;
     private String driver;
     private String odbcTableTypeName;
+    private String charset;
+    private String extraParam;
+    private Map<String, String> resourceProperties;
 
     public OdbcTable() {
         super(TableType.ODBC);
@@ -107,7 +115,6 @@ private void validate(Map<String, String> properties) throws DdlException {
                     + "they are: odbc_catalog_resource or [host, port, user, password, driver, odbc_type]" +
                     " and database and table");
         }
-
         if (properties.containsKey(ODBC_CATALOG_RESOURCE)) {
             odbcCatalogResourceName = properties.get(ODBC_CATALOG_RESOURCE);
 
@@ -125,13 +132,26 @@ private void validate(Map<String, String> properties) throws DdlException {
                         + "'@'" + ConnectContext.get().getRemoteIP()
                         + "' for resource '" + odbcCatalogResourceName + "'");
             }
+            resourceProperties = new HashMap<>(oriResource.getCopiedProperties());
+            resourceProperties.remove(ODBC_HOST);
+            resourceProperties.remove(ODBC_PORT);
+            resourceProperties.remove(ODBC_USER);
+            resourceProperties.remove(ODBC_PASSWORD);
+            resourceProperties.remove(ODBC_DRIVER);
+            resourceProperties.remove(ODBC_CHARSET);
+            resourceProperties.remove(ODBC_TYPE);
+            resourceProperties.remove("type");
+            resourceProperties.remove(ODBC_DATABASE);
         } else {
+            Map<String, String> copiedProperties = new HashMap<>();
+            copiedProperties.putAll(properties);
             // Set up
             host = properties.get(ODBC_HOST);
             if (Strings.isNullOrEmpty(host)) {
                 throw new DdlException("Host of Odbc table is null. "
                         + "Please set proper resource or add properties('host'='xxx.xxx.xxx.xxx') when create table");
             }
+            copiedProperties.remove(ODBC_HOST);
 
             port = properties.get(ODBC_PORT);
             if (Strings.isNullOrEmpty(port)) {
@@ -147,25 +167,33 @@ private void validate(Map<String, String> properties) throws DdlException {
 
                 }
             }
+            copiedProperties.remove(ODBC_PORT);
 
             userName = properties.get(ODBC_USER);
             if (Strings.isNullOrEmpty(userName)) {
                 throw new DdlException("User of Odbc table is null. "
                         + "Please set odbc_catalog_resource or add properties('user'='root') when create table");
             }
+            copiedProperties.remove(ODBC_USER);
 
             passwd = properties.get(ODBC_PASSWORD);
             if (passwd == null) {
                 throw new DdlException("Password of Odbc table is null. "
                         + "Please set odbc_catalog_resource or add properties('password'='xxxx') when create table");
             }
+            copiedProperties.remove(ODBC_PASSWORD);
 
             driver = properties.get(ODBC_DRIVER);
             if (Strings.isNullOrEmpty(driver)) {
                 throw new DdlException("Driver of Odbc table is null. "
                         + "Please set odbc_catalog_resource or add properties('diver'='xxxx') when create table");
             }
+            copiedProperties.remove(ODBC_DRIVER);
+
 
+            charset = properties.get(ODBC_CHARSET);
+            copiedProperties.remove(ODBC_CHARSET);
+
             String tableType = properties.get(ODBC_TYPE);
             if (Strings.isNullOrEmpty(tableType)) {
                 throw new DdlException("Type of Odbc table is null. "
@@ -177,6 +205,10 @@ private void validate(Map<String, String> properties) throws DdlException {
                             + " Now Odbc table type only support:" + supportTableType());
                 }
             }
+            copiedProperties.remove(ODBC_TYPE);
+            copiedProperties.remove(ODBC_DATABASE);
+            copiedProperties.remove(ODBC_TABLE);
+            extraParam = getExtraParameter(copiedProperties);
         }
 
         odbcDatabaseName = properties.get(ODBC_DATABASE);
@@ -187,7 +219,7 @@ private void validate(Map<String, String> properties) throws DdlException {
 
         odbcTableName = properties.get(ODBC_TABLE);
         if (Strings.isNullOrEmpty(odbcTableName)) {
-            throw new DdlException("Database of Odbc table is null. "
+            throw new DdlException("Table of Odbc table is null. "
                     + "Please add properties('table'='xxxx') when create table");
         }
     }
@@ -199,13 +231,28 @@ private String getPropertyFromResource(String propertyName) {
             throw new RuntimeException("Resource does not exist. name: " + odbcCatalogResourceName);
         }
 
-        String property = odbcCatalogResource.getProperties(propertyName);
+        String property = odbcCatalogResource.getProperty(propertyName);
         if (property == null) {
             throw new RuntimeException("The property:" + propertyName + " do not set in resource " + odbcCatalogResourceName);
         }
         return property;
     }
+    public String getExtraParameter(Map<String, String> extraMap) {
+        if (extraMap == null || extraMap.isEmpty()) {
+            return "";
+        }
+        return ";" + extraMap.entrySet()
+                .stream()
+                .map(e -> e.getKey() + "=" + e.getValue())
+                .collect(joining(";"));
+    }
 
+    public String getExtraParam() {
+        if (extraParam != null) {
+            return extraParam;
+        }
+        return getExtraParameter(resourceProperties);
+    }
     public String getOdbcCatalogResourceName() {
         return odbcCatalogResourceName;
     }
@@ -253,6 +300,20 @@ public String getOdbcDriver() {
         return getPropertyFromResource(ODBC_DRIVER);
     }
 
+    public String getCharset() {
+        if (charset != null) {
+            return charset;
+        }
+        String resourceCharset = "utf8";
+        try {
+            resourceCharset = getPropertyFromResource(ODBC_CHARSET);
+        } catch (Exception e) {
+            LOG.info(e.getMessage());
+        }
+
+        return resourceCharset;
+    }
+
     public String getOdbcTableTypeName() {
         if (odbcTableTypeName != null) {
             return odbcTableTypeName;
@@ -273,7 +334,7 @@ public String getConnectString() {
                         getOdbcDatabaseName(),
                         getUserName(),
                         getPasswd(),
-                        "utf8");
+                        getCharset());
                 break;
             case POSTGRESQL:
                 connectString = String.format("Driver=%s;Server=%s;Port=%s;DataBase=%s;Uid=%s;Pwd=%s;charset=%s;UseDeclareFetch=1;Fetch=4096",
@@ -283,7 +344,7 @@ public String getConnectString() {
                         getOdbcDatabaseName(),
                         getUserName(),
                         getPasswd(),
-                        "utf8");
+                        getCharset());
                 break;
             case MYSQL:
                 connectString = String.format("Driver=%s;Server=%s;Port=%s;DataBase=%s;Uid=%s;Pwd=%s;charset=%s;forward_cursor=1;no_cache=1",
@@ -293,20 +354,20 @@ public String getConnectString() {
                         getOdbcDatabaseName(),
                         getUserName(),
                         getPasswd(),
-                        "utf8");
+                        getCharset());
                 break;
             case SQLSERVER:
                 connectString = String.format("Driver=%s;Server=%s,%s;DataBase=%s;Uid=%s;Pwd=%s",
-                                getOdbcDriver(),
-                                getHost(),
-                                getPort(),
-                                getOdbcDatabaseName(),
-                                getUserName(),
-                                getPasswd());
+                        getOdbcDriver(),
+                        getHost(),
+                        getPort(),
+                        getOdbcDatabaseName(),
+                        getUserName(),
+                        getPasswd());
                 break;
             default:
         }
-        return connectString;
+        return connectString + getExtraParam();
     }
 
     public TOdbcTableType getOdbcTableType() {
@@ -361,6 +422,8 @@ public String getSignature(int signatureVersion) {
             sb.append(passwd);
             sb.append(driver);
             sb.append(odbcTableTypeName);
+            sb.append(charset);
+            sb.append(extraParam);
         }
         String md5 = DigestUtils.md5Hex(sb.toString());
         LOG.debug("get signature of odbc table {}: {}. signature string: {}", name, md5, sb.toString());
@@ -382,6 +445,8 @@ public void write(DataOutput out) throws IOException {
         serializeMap.put(ODBC_TABLE, odbcTableName);
         serializeMap.put(ODBC_DRIVER, driver);
         serializeMap.put(ODBC_TYPE, odbcTableTypeName);
+        serializeMap.put(ODBC_CHARSET, charset);
+        serializeMap.put(ODBC_EXTRA_PARAM, extraParam);
 
         int size = (int) serializeMap.values().stream().filter(v -> {
             return v != null;
@@ -417,6 +482,8 @@ public void readFields(DataInput in) throws IOException {
         odbcTableName = serializeMap.get(ODBC_TABLE);
         driver = serializeMap.get(ODBC_DRIVER);
         odbcTableTypeName = serializeMap.get(ODBC_TYPE);
+        charset = serializeMap.get(ODBC_CHARSET);
+        extraParam = serializeMap.get(ODBC_EXTRA_PARAM);
     }
 
     public static String supportTableType() {

fe/fe-core/src/main/java/org/apache/doris/catalog/Resource.java

@@ -136,6 +136,8 @@ protected void replaceIfEffectiveValue(Map<String, String> properties, String ke
      */
     protected abstract void setProperties(Map<String, String> properties) throws DdlException;
 
+
+    public abstract Map<String, String> getCopiedProperties();
     /**
      * Fill BaseProcResult with different properties in child resources
      * ResourceMgr.RESOURCE_PROC_NODE_TITLE_NAMES format:

fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java

@@ -139,6 +139,12 @@ public void checkProperties(Map<String, String> properties) throws AnalysisExcep
         }
     }
 
+    @Override
+    public Map<String, String> getCopiedProperties() {
+        Map<String, String> copiedProperties = Maps.newHashMap(properties);
+        return copiedProperties;
+    }
+
     @Override
     protected void getProcNodeData(BaseProcResult result) {
         String lowerCaseType = type.name().toLowerCase();

fe/fe-core/src/main/java/org/apache/doris/catalog/SparkResource.java

@@ -158,6 +158,11 @@ public SparkResource getCopiedResource() {
         return new SparkResource(name, Maps.newHashMap(sparkConfigs), workingDir, broker, brokerProperties);
     }
 
+    @Override
+    public Map<String, String> getCopiedProperties() {
+        Map<String, String> copiedProperties = Maps.newHashMap(sparkConfigs);
+        return copiedProperties;
+    }
     // Each SparkResource has and only has one SparkRepository.
     // This method get the remote archive which matches the dpp version from remote repository
     public synchronized SparkRepository.SparkArchive prepareArchive() throws LoadException {