-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SUPPORT] HoodieStreamer: HoodieKeyException error when using SQL Transformer #11635
Comments
Try |
Hi @wombatu-kun, Thank you for getting back to me. I have modified the SQL File to the following:
I'm still getting the same error:
Not sure if I'm missing something else. Thank you |
what if you try this?
|
Trying the suggested SQL is giving the following error:
|
Following up on this, I was able to run the spark app without errors, however, even when specifying columns that are in the 'before' and 'after' Struct type extracted from Kafka, only the following columns were written to the Hudi table:
It seems the additional columns extracted from the envelopes 'before' and 'after' are being ignored for some reason. I even tried to specify a static columns like: 'test' AS test_columns, it was not written on the Hudi tables.
Is there something else that I might be missing? Thank you |
As I see from documentation here https://hudi.apache.org/docs/hoodie_streaming_ingestion#schema-providers
May be you should add target schema with additional fields to schema registry and provide Url to that schema in config |
@wombatu-kun Couldn't that be extracted from the Transformer? As I understand it, the Transformer sits between the Source and Target. For some reason my additional fields are being ignored. Thank you |
Hi @wombatu-kun, I wanted to provide an update on the issue. After some experimentation with the Thank you |
Hello,
We are using
org.apache.hudi.utilities.streamer.HoodieStreamer
class to extract data out of Kafka and write to Hudi table. The Kafka topic is populated via Debezium using SQL Server table. We are using EMR 7.1.0 to run Spark-Submit.To Reproduce
Steps to reproduce the behavior:
Environment Description
Hudi version : 0.15
Spark version : 3.5.0
Hive version : N/A
Hadoop version : N/A
Storage : S3
Running on Docker? : No
Error Message
I have confirmed that the transformer SQL file is extracting the data correctly. Could you please verify whether the transformer runs before the Hudi writer and if it is supplying the writer with the final transformed data? If so, could you help determine why the 'hudi_key' field is not being passed correctly? I can confirm that the 'hudi_key' is generated by concatenating two non-null fields.
Thank you
The text was updated successfully, but these errors were encountered: