JDBC java.lang.NoSuchFieldError: INSTANCE

[HadoopChen]

命令

hadoop_lt jar clickhouse-hdfs-loader-2.0.3-jar-with-dependencies.jar com.kugou.loader.clickhouse.ClickhouseHdfsLoader
-Dmapreduce.job.queuename=root.cj.default
-i orc
--connect jdbc:clickhouse:/1.2.3.4:8123/default
--table distributed_tag
--dt 2018-06-23
--export-dir /user/cj/hive_db/cj.db/tag2

错误

18/06/23 17:47:19 INFO clickhouse.ClickhouseClientHolder: Clickhouse Loader : get clickhouse client[jdbc:clickhouse://1.2.3.4:8123/default] for user=null
18/06/23 17:47:19 INFO clickhouse.ClickHouseDriver: Driver registered
18/06/23 17:47:19 INFO clickhouse.ClickHouseDriver: Creating connection
Exception in thread "main" java.lang.NoSuchFieldError: INSTANCE
at org.apache.http.impl.io.DefaultHttpRequestWriterFactory.(DefaultHttpRequestWriterFactory.java:52)
at org.apache.http.impl.io.DefaultHttpRequestWriterFactory.(DefaultHttpRequestWriterFactory.java:56)
at org.apache.http.impl.io.DefaultHttpRequestWriterFactory.(DefaultHttpRequestWriterFactory.java:46)
at org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.(ManagedHttpClientConnectionFactory.java:82)
at org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.(ManagedHttpClientConnectionFactory.java:95)
at org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.(ManagedHttpClientConnectionFactory.java:104)
at org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.(ManagedHttpClientConnectionFactory.java:62)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.(PoolingHttpClientConnectionManager.java:572)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.(PoolingHttpClientConnectionManager.java:174)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.(PoolingHttpClientConnectionManager.java:158)
at ru.yandex.clickhouse.util.ClickHouseHttpClientBuilder.getConnectionManager(ClickHouseHttpClientBuilder.java:77)
at ru.yandex.clickhouse.util.ClickHouseHttpClientBuilder.buildClient(ClickHouseHttpClientBuilder.java:54)
at ru.yandex.clickhouse.ClickHouseConnectionImpl.(ClickHouseConnectionImpl.java:71)
at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:58)
at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:50)
at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:32)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:233)
at com.kugou.loader.clickhouse.ClickhouseClient.(ClickhouseClient.java:47)
at com.kugou.loader.clickhouse.ClickhouseClient.(ClickhouseClient.java:39)
at com.kugou.loader.clickhouse.ClickhouseClientHolder.getClickhouseClient(ClickhouseClientHolder.java:49)
at com.kugou.loader.clickhouse.ClickhouseClientHolder.getClickhouseClient(ClickhouseClientHolder.java:57)
at com.kugou.loader.clickhouse.ClickhouseHdfsLoader.initClickhouseParameters(ClickhouseHdfsLoader.java:223)
at com.kugou.loader.clickhouse.ClickhouseHdfsLoader.run(ClickhouseHdfsLoader.java:121)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at com.kugou.loader.clickhouse.ClickhouseHdfsLoader.main(ClickhouseHdfsLoader.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

排查

telnet 1.2.3.4 8123 ==> OK
default数据库存在、分布式表distributed_tag也存在；
clickhouseServer正常，无报错日志；

求助！！！

[jaykelin]

缺少参数：--username xxx --password xxx

[HadoopChen]

哦哦 感谢！

[HadoopChen]

貌似依然不行

CH本机可以通过clickhouse-client --user default --password default登入CH；但使用JDBC依然不行；报同样的错；均是默认配置，只是给default用户加了一个明文的密码。

<password>default</password>
<networks incl="networks" replace="replace">
               <ip>::/0</ip>
</networks>
<profile>default</profile>
<quota>default</quota>
</default>

命令

hadoop_lt jar clickhouse-hdfs-loader-2.0.3-jar-with-dependencies.jar com.kugou.loader.clickhouse.ClickhouseHdfsLoader
-Dmapreduce.job.queuename=root.cj.default
-i orc
--connect jdbc:clickhouse:/1.2.3.4:8123/default
--username default --password default
--table distributed_tag
--dt 2018-06-23
--export-dir /user/cj/hive_db/cj.db/tag2

错误

18/06/26 10:35:06 INFO clickhouse.ClickhouseClientHolder: Clickhouse Loader : get clickhouse client[jdbc:clickhouse://1.2.3.4:8123/default] for user=default
18/06/26 10:35:06 INFO clickhouse.ClickHouseDriver: Driver registered
18/06/26 10:35:06 INFO clickhouse.ClickHouseDriver: Creating connection
Exception in thread "main" java.lang.NoSuchFieldError: INSTANCE
at org.apache.http.impl.io.DefaultHttpRequestWriterFactory.(DefaultHttpRequestWriterFactory.java:52)
at org.apache.http.impl.io.DefaultHttpRequestWriterFactory.(DefaultHttpRequestWriterFactory.java:56)
at org.apache.http.impl.io.DefaultHttpRequestWriterFactory.(DefaultHttpRequestWriterFactory.java:46)
at org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.(ManagedHttpClientConnectionFactory.java:82)
at org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.(ManagedHttpClientConnectionFactory.java:95)
at org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.(ManagedHttpClientConnectionFactory.java:104)
at org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.(ManagedHttpClientConnectionFactory.java:62)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.(PoolingHttpClientConnectionManager.java:572)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.(PoolingHttpClientConnectionManager.java:174)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.(PoolingHttpClientConnectionManager.java:158)
at ru.yandex.clickhouse.util.ClickHouseHttpClientBuilder.getConnectionManager(ClickHouseHttpClientBuilder.java:77)
at ru.yandex.clickhouse.util.ClickHouseHttpClientBuilder.buildClient(ClickHouseHttpClientBuilder.java:54)
at ru.yandex.clickhouse.ClickHouseConnectionImpl.(ClickHouseConnectionImpl.java:71)
at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:58)
at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:50)
at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:32)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:215)
at com.kugou.loader.clickhouse.ClickhouseClient.(ClickhouseClient.java:45)
at com.kugou.loader.clickhouse.ClickhouseClientHolder.getClickhouseClient(ClickhouseClientHolder.java:65)
at com.kugou.loader.clickhouse.ClickhouseHdfsLoader.initClickhouseParameters(ClickhouseHdfsLoader.java:223)
at com.kugou.loader.clickhouse.ClickhouseHdfsLoader.run(ClickhouseHdfsLoader.java:121)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at com.kugou.loader.clickhouse.ClickhouseHdfsLoader.main(ClickhouseHdfsLoader.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

[jaykelin]

@HadoopChen http客户端的jar包是不是有冲突

[HadoopChen]

• CH依赖于httpclient 4.5.2，httpcore 4.5.2
• hadoop集群2.5.2之后hadoop-common就依赖于httpcore+httpclient 4.5.5，集群环境的jar包无法更换；
• CH JDBC里面用的HttpClientConnectionManager是属于4.5.2的，如果用provided，则提示不存在该类；如果去掉provided，用4.5.2的版本，又和集群的4.5.5冲突。

咋搞，把jdbc用4.5.5改一遍。旧的类都替换掉 ？

[jaykelin]

试试用这个jdbc客户端
https://github.com/jaykelin/clickhouse-jdbc/tree/0.1-SNAPSHOT

[panda403]

username 和password是clickhouse的用户名密码吗？我默认单机装的没有用户名密码怎么弄

[slimtom95]

use shade relocation to solve this conflicts ref to

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.2.1</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <relocations>
                    <relocation>
                        <pattern>org.apache.http</pattern>
                        <shadedPattern>shaded.org.apache.http</shadedPattern>
                    </relocation>
                </relocations>
            </configuration>
        </execution>
    </executions>
</plugin>

[Balro]

将冲突包设置为compile，提交作业前设置变量调整hadoop类路径加载顺序，后提交作业。
export HADOOP_CLASSPATH=xxx.jar
export HADOOP_USER_CLASSPATH_FIRST=true
提交作业时，再加参 -Dmapreduce.job.user.classpath.first=true