Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Refactored JSON IO (better support for JSON and NDJSON) #870

Merged
merged 6 commits into from Feb 27, 2022
Merged

Conversation

jorgecarleitao
Copy link
Owner

This PR refactors APIs to read and write to JSON and NDJSON.

Currently, writing a primitive array to JSON requires writing a list of objects (e.g. [{"a": 1}, {"a": null}]). However, JSON supports simpler representations of a primitive array, [1, null].

The same holds true for NDJSON, where we required the user to write

{"a": 1}
{"a": null}

This PR removes this, allowing users to write [1, null] and 1\nnull respectively. The previous behavior is now a particular case, via StructArray.

The APIs of json and ndjson were also separated in two distinct modules since they are sufficiently different formats. All the public API changes are shown in the changes to the examples, and the guide was updated accordingly.

@jorgecarleitao jorgecarleitao added the enhancement An improvement to an existing feature label Feb 27, 2022
@codecov
Copy link

codecov bot commented Feb 27, 2022

Codecov Report

Merging #870 (1539556) into main (fca3b8d) will decrease coverage by 0.01%.
The diff coverage is 90.07%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #870      +/-   ##
==========================================
- Coverage   71.52%   71.50%   -0.02%     
==========================================
  Files         335      335              
  Lines       17989    17950      -39     
==========================================
- Hits        12867    12836      -31     
+ Misses       5122     5114       -8     
Impacted Files Coverage Δ
src/io/json/read/deserialize.rs 66.88% <75.00%> (-1.95%) ⬇️
src/io/ndjson/read/file.rs 78.57% <78.57%> (ø)
src/io/json/write/serialize.rs 85.38% <90.90%> (-0.73%) ⬇️
src/io/json/write/mod.rs 95.00% <92.85%> (-5.00%) ⬇️
src/io/json/read/infer_schema.rs 91.25% <100.00%> (+4.76%) ⬆️
src/io/ndjson/read/deserialize.rs 100.00% <100.00%> (ø)
src/io/ndjson/write/mod.rs 100.00% <100.00%> (ø)
src/error.rs 38.46% <0.00%> (-7.70%) ⬇️
src/bitmap/utils/slice_iterator.rs 87.93% <0.00%> (+1.72%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fca3b8d...1539556. Read the comment docs.

@jorgecarleitao jorgecarleitao merged commit e545ae5 into main Feb 27, 2022
@jorgecarleitao jorgecarleitao deleted the ndjson branch February 27, 2022 21:35
@jorgecarleitao jorgecarleitao removed the enhancement An improvement to an existing feature label Mar 6, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant