-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SUPPORT] BQ synch tool not working with HUDI bundle jar #10629
Comments
@masthanmca Your configurations also looks wrong? From where you got these or which doc you referred? |
Facing the same issue , does not work with org.apache.hudi:hudi-spark3.3-bundle_2.12:0.14.1 . Hudi Write to path works , Hive Sync works but BQ sync does not work. For now have taken this route based on a flag to manually perform the BQSync with BQSyncTool post the dataframe.write |
@abhishekshenoy @masthanmca That (#9355 (comment)) i.e. BigQuerySyncTool is the correct way of doing BQ sync with batch jobs. The another way is doing this with HudiStreamer. |
@ad1happy2go @the-other-tim-brown
|
@masthanmca @abhishekshenoy I went through the code and identified that we need to set both the class names to do both metastync together. The default value for below prop is just hive sync. I tried with 0.14.1 hudi version and after write and hive sync completed, it tried to do Big query sync also.
|
@masthanmca Closing out this issue as I confirmed it works. Please reopen in case you still see this issue. |
Tips before filing an issue
Have you gone through our FAQs? yes
Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
If you have triaged this as a bug, then file an issue directly.
Describe the problem you faced
BQ sync is not working with hudi bundle jar
A clear and concise description of the problem.
I wanted to enable BQ sync while writing ingest the data into HUDI table using manifest file.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Environment Description
Hudi version : 0.14.0
Spark version : 3.3.2
Hive version :
Hadoop version :
Storage (HDFS/S3/GCS..) : GCS
Running on Docker? (yes/no) :no
Additional context
Add any other context about the problem here.
Stacktrace
Add the stacktrace of the error.
No error , but external table not created in Big Query
The text was updated successfully, but these errors were encountered: