-
Notifications
You must be signed in to change notification settings - Fork 374
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: replace deprecated arrow::json::reader::Decoder #1226
fix: replace deprecated arrow::json::reader::Decoder #1226
Conversation
The problem appears to be the |
I created a PR to add the kernel upstream. |
53a4fe2
to
ddf70f1
Compare
Now looking at these clippy errors 😒 |
This will be addressed by #1249 , so once that pull request is ready, this test should be merged into that branch |
ddf70f1
to
63e7d9f
Compare
63e7d9f
to
e36a074
Compare
ACTION NEEDED delta-rs follows the Conventional Commits The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. |
e36a074
to
4c43eb9
Compare
3ebf6be
to
12c02a8
Compare
3629d55
to
4265844
Compare
4c2ce43
to
4265844
Compare
@@ -9,7 +11,7 @@ def test_table_schema(): | |||
table_path = "../rust/tests/data/simple_table" | |||
dt = DeltaTable(table_path) | |||
schema = dt.schema() | |||
assert schema.json() == { | |||
assert json.loads(schema.to_json()) == { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm confused. This seems like an API-breaking change? What happened? nevermind I can't read. Looks good 👍
rust/src/writer/utils.rs
Outdated
let mut buf = vec![]; | ||
for message in message_buffer { | ||
buf.write_all( | ||
serde_json::to_string(&message) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
They are making you re-serialize the messages?! Ouch...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did you consider this API? https://docs.rs/arrow-json/39.0.0/arrow_json/reader/struct.Decoder.html#method.serialize
That's what Raphael built for the use case you were asking about, I think. It avoids the overhead of serializing the data to the buffer.
This commit incorporates the latest upstream error changes and ensures that a RecordBatch which contains map data _with_ a partition can be segmented out correctly. This depends on the upstream work to support take() on MapArray that @wjones127 recently committed
…JSON is not supported
667dbea
to
cf881f5
Compare
…deprecated in arrow 38
cf881f5
to
aecd898
Compare
rust/src/checkpoints.rs
Outdated
let mut buf = vec![]; | ||
for res in jsons { | ||
let json = res?; | ||
buf.write_all(serde_json::to_string(&json)?.as_bytes())?; | ||
} | ||
let mut consumed = 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same thing here, right? we shouldn't need to serialize to json string if in the end we are trying to make a parquet file?
Description
This test will fail with the following error:
As far as I can tell this should be supported behavior and a supported schema. Note that with no partition columns the write works just fine.
=# Related Issue(s)
apache/arrow-rs#3875