Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Carbondata writer 看现在用的1.5的版本, 会向下兼容carbondata1.3版本吗 #56

Closed
HeineG opened this issue May 15, 2019 · 4 comments

Comments

@HeineG
Copy link

HeineG commented May 15, 2019

新手carbondata, 不知道写carbondata文件结构是怎样的,下载下来测试异常:
java.lang.RuntimeException: java.lang.InterruptedException: org.apache.carbondata.core.datastore.exception.CarbonDataWriterException: org.apache.carbondata.core.datastore.exception.CarbonDataWriterException
at com.dtstack.flinkx.carbondata.writer.CarbonOutputFormat.closeInternal(CarbonOutputFormat.java:261)
at com.dtstack.flinkx.outputformat.RichOutputFormat.close(RichOutputFormat.java:346)
at org.apache.flink.streaming.api.functions.sink.OutputFormatSinkFunction.close(OutputFormatSinkFunction.java:96)
at org.apache.flink.api.common.functions.util.FunctionUtils.closeFunction(FunctionUtils.java:43)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.close(AbstractUdfStreamOperator.java:109)
at org.apache.flink.streaming.runtime.tasks.StreamTask.closeAllOperators(StreamTask.java:438)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:323)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:712)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.InterruptedException: org.apache.carbondata.core.datastore.exception.CarbonDataWriterException: org.apache.carbondata.core.datastore.exception.CarbonDataWriterException
at org.apache.carbondata.hadoop.api.CarbonTableOutputFormat$CarbonRecordWriter.close(CarbonTableOutputFormat.java:456)
at com.dtstack.flinkx.carbondata.writer.recordwriter.AbstractRecordWriter.closeRecordWriter(AbstractRecordWriter.java:108)
at com.dtstack.flinkx.carbondata.writer.recordwriter.AbstractRecordWriter.close(AbstractRecordWriter.java:174)
at com.dtstack.flinkx.carbondata.writer.CarbonOutputFormat.closeInternal(CarbonOutputFormat.java:259)
... 8 more

@lijiangbo
Copy link
Contributor

新手carbondata, 不知道写carbondata文件结构是怎样的,下载下来测试异常:
java.lang.RuntimeException: java.lang.InterruptedException: org.apache.carbondata.core.datastore.exception.CarbonDataWriterException: org.apache.carbondata.core.datastore.exception.CarbonDataWriterException
at com.dtstack.flinkx.carbondata.writer.CarbonOutputFormat.closeInternal(CarbonOutputFormat.java:261)
at com.dtstack.flinkx.outputformat.RichOutputFormat.close(RichOutputFormat.java:346)
at org.apache.flink.streaming.api.functions.sink.OutputFormatSinkFunction.close(OutputFormatSinkFunction.java:96)
at org.apache.flink.api.common.functions.util.FunctionUtils.closeFunction(FunctionUtils.java:43)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.close(AbstractUdfStreamOperator.java:109)
at org.apache.flink.streaming.runtime.tasks.StreamTask.closeAllOperators(StreamTask.java:438)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:323)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:712)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.InterruptedException: org.apache.carbondata.core.datastore.exception.CarbonDataWriterException: org.apache.carbondata.core.datastore.exception.CarbonDataWriterException
at org.apache.carbondata.hadoop.api.CarbonTableOutputFormat$CarbonRecordWriter.close(CarbonTableOutputFormat.java:456)
at com.dtstack.flinkx.carbondata.writer.recordwriter.AbstractRecordWriter.closeRecordWriter(AbstractRecordWriter.java:108)
at com.dtstack.flinkx.carbondata.writer.recordwriter.AbstractRecordWriter.close(AbstractRecordWriter.java:174)
at com.dtstack.flinkx.carbondata.writer.CarbonOutputFormat.closeInternal(CarbonOutputFormat.java:259)
... 8 more

麻烦发下你的任务json,还有,错误日志只有这些吗?

@HeineG
Copy link
Author

HeineG commented May 15, 2019

debug进去这个异常, 看不懂,大神请教一下:

image

@yangsishu
Copy link
Contributor

carbondata1.3版本 没测试过

@HeineG
Copy link
Author

HeineG commented May 20, 2019

多谢回复,有个问题请教一下:
我用了一个环境是1.5.2版本的, 执行成功,文件写上去了
image
但是用spark shell去查询的时候,查询不出数据, 我看了代码执行是走hive区分那个类,请问查询是可以在哪里设置查询数据的类型吗?谢谢!
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants