You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently SPyQL only allows to read and write JSON lines. Writing JSON arrays can be done using the dict_agg, aggregating everything into an array and writing an JSON with a single line.
The idea is to add an argument lines=True to json and orjson writers and processors. The processor should be able to handle single object files as well as arrays of objects or arrays of scalars. When lines is False the processor should load the full input into memory and then parse it. While this is not ideal, it is the most straightforward implementation. In addition, arrays of JSON shouldn't be used for large data, in that cases JSON lines should be used instead.
The writer should write an array of objects when lines is False.
The text was updated successfully, but these errors were encountered:
Currently SPyQL only allows to read and write JSON lines. Writing JSON arrays can be done using the
dict_agg
, aggregating everything into an array and writing an JSON with a single line.The idea is to add an argument
lines=True
to json and orjson writers and processors. The processor should be able to handle single object files as well as arrays of objects or arrays of scalars. Whenlines
isFalse
the processor should load the full input into memory and then parse it. While this is not ideal, it is the most straightforward implementation. In addition, arrays of JSON shouldn't be used for large data, in that cases JSON lines should be used instead.The writer should write an array of objects when
lines
isFalse
.The text was updated successfully, but these errors were encountered: