-
-
Notifications
You must be signed in to change notification settings - Fork 73
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Querying set of CSVs which a lot of entries does not output any json #101
Comments
I want more information. Does trdsql exit without outputting anything (errors, etc.) in JSON? |
CSV output works fine, with or without LIMIT. JSON output works with a LIMIT (eg. 10), but not without LIMIT JSON without LIMIT gives no output at all, not even error messages. Tried also with debug flag, but no extra info is returned. |
Hmm ...
|
Saw that you are running on linux and realized that it is a memory issue. Pure from an install on Mac, it works and more than 12GB is consumed to produce the json. Running inside docker with more resources works also fine. Strange that docker gave no error message. Looking at the code (not a go dev myself), I suppose you are building everything in memory first and then write it to disk, and not streaming the data? |
I see. understood.
Yes. For example [{"id":"1","name":"Orange"},
{"id":"2","name":"Melon"},
{"id":"3","name":"Apple"}] For large outputs, you will need an output that is not an array (I don't know what to call this). {"id":"1","name":"Orange"}
{"id":"2","name":"Melon"}
{"id":"3","name":"Apple"} Not yet. I will consider it from now on. |
Add JSONL output format separately from JSON. JSONL (http://jsonlines.org/) is a JSON format separated by line breaks. Output can be output line by line, so output starts quickly. Also, JSONL output requires less memory than JSON output. resolve #101.
I added a new JSONL output option ( #103 ). |
I am working on a medium-large open data set of Belgian companies, which is freely downloadable at https://kbopub.economie.fgov.be/kbo-open-data/login
I am trying to execute following query to denormalize data into 1 file and it works fine when the output target is CSV, but nothing happens when outputting JSON:
Any ideas?
The text was updated successfully, but these errors were encountered: