Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use dbt-hive connection hive Error reporting "Unable to establish connection to Hive server: Error while compiling statement: FAILED: ParseException line 1:0 cannot recognize input near '/' '*' '{'" #129

Closed
1043038630 opened this issue Sep 22, 2023 · 4 comments

Comments

@1043038630
Copy link

Using unsafe methods:
profiles.yml:
dbt_hive_test:
outputs:
dev:
type: hive
host: 192.168.1.143
port: 10000
schema: "default"
target: dev

Error running dbt

(dbt-hive-test) [root@localhost dbt_hive_test]# dbt run
09:00:34  Running with dbt=1.4.8
09:00:34  Unable to do partial parsing because profile has changed
09:00:36  Found 2 models, 0 tests, 0 snapshots, 0 analyses, 334 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics
09:00:36  
09:00:37  
09:00:37  Finished running  in 0 hours 0 minutes and 1.04 seconds (1.04s).
09:00:38  Encountered an error:
Runtime Error
  Runtime Error
    Unable to establish connection to Hive server: Error while compiling statement: FAILED: ParseException line 1:0 cannot recognize input near '/' '*' '{'
(dbt-hive-test) [root@localhost dbt_hive_test]# dbt run
09:04:16  Running with dbt=1.4.8
09:04:16  Found 2 models, 0 tests, 0 snapshots, 0 analyses, 334 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics
09:04:16  
09:04:17  
09:04:17  Finished running  in 0 hours 0 minutes and 0.75 seconds (0.75s).
09:04:18  Encountered an error:
Runtime Error
  Runtime Error
    Unable to establish connection to Hive server: Error while compiling statement: FAILED: ParseException line 1:0 cannot recognize input near '/' '*' '{'

Use safe methods:
profiles.yml:
dbt_hive_test:
outputs:
dev:
type: hive
host: 192.168.1.143
port: 10000
auth_type : LDAP
use_http_transport: false
use_ssl: false
user : "uid=query,ou=People,dc=pudu,dc=com"
password : "000000"
schema: "default"
target: dev

I confirm that the above method can link to hive,I tested it with beeline。

[root@cdhmaster bin]# beeline --color=true -u "jdbc:hive2://cdhslave2:10000/default" -n uid=query,ou=People,dc=pudu,dc=com -p 000000
Connecting to jdbc:hive2://cdhslave2:10000/default
Connected to: Apache Hive (version 2.1.1-cdh6.3.2)
Driver: Hive JDBC (version 2.1.1-cdh6.3.2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 2.1.1-cdh6.3.2 by Apache Hive

Testing with dbt is also normal

(dbt-hive-test) [root@localhost dbt_hive_test]# dbt debug
08:59:39  Running with dbt=1.4.8
dbt version: 1.4.8
python version: 3.9.17
python path: /root/miniconda3/bin/python
os info: Linux-3.10.0-1160.92.1.el7.x86_64-x86_64-with-glibc2.17
Using profiles.yml file at /root/.dbt/profiles.yml
Using dbt_project.yml file at /root/gitlab/dbt/dbt-cdh6.3.2-test/dbt_hive_test/dbt_project.yml

Configuration:
  profiles.yml file [OK found and valid]
  dbt_project.yml file [OK found and valid]

Required dependencies:
 - git [OK found]

Connection:
  host: 192.168.1.143
  schema: default
  user: uid=query,ou=People,dc=pudu,dc=com
  Connection test: [OK connection ok]

All checks passed!

But there will be errors when running

(dbt-hive-test) [root@localhost dbt_hive_test]# dbt run
09:04:16  Running with dbt=1.4.8
09:04:16  Found 2 models, 0 tests, 0 snapshots, 0 analyses, 334 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics
09:04:16  
09:04:17  
09:04:17  Finished running  in 0 hours 0 minutes and 0.75 seconds (0.75s).
09:04:18  Encountered an error:
Runtime Error
  Runtime Error
    Unable to establish connection to Hive server: Error while compiling statement: FAILED: ParseException line 1:0 cannot recognize input near '/' '*' '{'

I checked the logs of hive JDBC,It is speculated that the SQL converted from dbt has added annotations that Hive does not recognize,But I'm not sure /* */

2023-09-22 16:17:12,456 INFO  org.apache.hadoop.hive.ql.Driver: [202ed937-5191-4d28-934a-341c95b622af HiveServer2-Handler-Pool: Thread-56]: Compiling command(queryId=hive_20230922161712_ef9241ae-634a-451e-a68d-8992a0932fa7): /* {"app": "dbt", "dbt_version": "1.4.8", "profile_name": "dbt_hive_test", "target_name": "dev", "connection_name": "list_schemas"} */

    show databases
  
2023-09-22 16:17:12,476 ERROR org.apache.hadoop.hive.ql.Driver: [202ed937-5191-4d28-934a-341c95b622af HiveServer2-Handler-Pool: Thread-56]: FAILED: ParseException line 1:0 cannot recognize input near '/' '*' '{'
org.apache.hadoop.hive.ql.parse.ParseException: line 1:0 cannot recognize input near '/' '*' '{'
	at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:221)
	at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:75)
	at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:68)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:564)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1425)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1398)
	at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:205)
	at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290)
	at org.apache.hive.service.cli.operation.Operation.run(Operation.java:260)
	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:505)
	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:491)
	at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:295)
	at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:507)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

@driftersxxs
Copy link

Is there a solution?

@1043038630
Copy link
Author

Is there a solution?

no 没有呢伙计 这个项目维护的人太少了 可以看看源码自己改改

@driftersxxs
Copy link

dat_project.yml 加上这个就可以了,去掉无效备注导致无法识别query-comment: null

@1043038630
Copy link
Author

dat_project.yml 加上这个就可以了,去掉无效备注导致无法识别query-comment: null

好的 谢谢, 应该是 dbt_project.yml 文件吧? 我在官网中只找到了 dbt_project.yml 的相关说明,以及dbt_project.yml文件里关于query-comment这个说明

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants