You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As part of #24616 we added JSON support, however, this has the problem of requiring us to buffer all of the data into memory. Given a sufficiently large query this would cause a possible OOM kill and so we should offer JSON Lines where each line is a row from the returned query. This would allow us to stream the data from a SendableRecordBatchStream as we get it, to the user, and avoid buffering too much data into memory.
Steps:
Create a JsonLinesFormatter in the influxdb3_server/src/http.rs file that uses the JsonFormat trait to set the output to follow JSON lines e.g. instead of the JSON:
[ { "foo": "bar" }, {"foo": "baz" } ]
it should be output as:
{ "foo": "bar" }
{ "foo": "baz" }
where each record is on it's own line without the []
This should just be writing a '\n' char into the writer in the end_row function and the rest should just need to be stubbed out with Ok(())
Using the Writer and the JsonLinesFormatter write the data into a streaming body in a separate task, while returning the Response for the API in the query routes
The text was updated successfully, but these errors were encountered:
As part of #24616 we added JSON support, however, this has the problem of requiring us to buffer all of the data into memory. Given a sufficiently large query this would cause a possible OOM kill and so we should offer JSON Lines where each line is a row from the returned query. This would allow us to stream the data from a
SendableRecordBatchStream
as we get it, to the user, and avoid buffering too much data into memory.Steps:
JsonLinesFormatter
in the influxdb3_server/src/http.rs file that uses theJsonFormat
trait to set the output to follow JSON lines e.g. instead of the JSON:it should be output as:
where each record is on it's own line without the
[]
This should just be writing a '\n' char into the writer in the
end_row
function and the rest should just need to be stubbed out withOk(())
Writer
and theJsonLinesFormatter
write the data into a streaming body in a separate task, while returning the Response for the API in thequery
routesThe text was updated successfully, but these errors were encountered: