Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support reading and writing of JSON objects (not JSON lines) #83

Open
dcmoura opened this issue Nov 17, 2022 · 1 comment
Open

Support reading and writing of JSON objects (not JSON lines) #83

dcmoura opened this issue Nov 17, 2022 · 1 comment
Labels
core Core feature

Comments

@dcmoura
Copy link
Owner

dcmoura commented Nov 17, 2022

Currently SPyQL only allows to read and write JSON lines. Writing JSON arrays can be done using the dict_agg, aggregating everything into an array and writing an JSON with a single line.

The idea is to add an argument lines=True to json and orjson writers and processors. The processor should be able to handle single object files as well as arrays of objects or arrays of scalars. When lines is False the processor should load the full input into memory and then parse it. While this is not ideal, it is the most straightforward implementation. In addition, arrays of JSON shouldn't be used for large data, in that cases JSON lines should be used instead.

The writer should write an array of objects when lines is False.

@dcmoura
Copy link
Owner Author

dcmoura commented Nov 17, 2022

BTW, reading JSON arrays is done today by leveraging jq in the command-line:

jq `.[]` myfile.json | spyql "SELECT .aproperty FROM json"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Core feature
Projects
None yet
Development

No branches or pull requests

1 participant