-
-
Notifications
You must be signed in to change notification settings - Fork 135
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Dasel query crashes with go panic #392
Comments
Hey @I-Al-Istannen, Thanks for raising this. I'll do a bit of digging. Out of interest, what were you actually trying to get from the CSV? |
It seems this panic could occur when appending to a list of objects. I have fixed it here: #393 however it requires the deletion of some code that I need to verify is OK to be removed. |
Thanks :)
I was parsing stop data for the German train network, which of course only offers data in a large CSV file :P The full command for finding stops I was interested in looked something like this: dasel -r csv -f <(iconv -f ISO-8859-1 -t UTF-8 haltestellen.csv) --csv-comma ';' -w json | jq 'map(select(.Gemeinde | contains("Name"))) | map({Gemeinde, Haltestelle, Haltestelle_lang, globaleID})' I was trying to do the select with |
Yeah dasel unfortunately loads the entire file into memory because of the change in file type. Streaming parsers are hard enough on there own - imagine trying to streaming readers + writers in 5 file types. Performance aside, I'd be interested in a subset of your file that I can create some tests against to: |
:^) You can find the full file for e.g. Baden-Württemberg here. They offer data in the only two capable data formats this world has come up with, semicolon-separated comma separated values and XLSX. You can follow the "csv Datei" link (or click here directly). The query is in my last comment, it just takes the json array, filters the entries to the ones in a specific municipality and then trims down the json object to the relevant elements. |
Hey @I-Al-Istannen, Just for your information I got some time to dig at this. This is possible without hitting the panic you experienced:
This shows just the fields:
For any stations in:
I have also merged the panic fix to master. |
Ah, I see. You need to also duplicate all the fields in the map as it is |
That's right - I want a neater version of that map function for exactly this purpose. It works for now though |
Describe the bug
dasel crashes with
panic: reflect.Set: value of type []interface {} is not assignable to type []*dencoding.Map
.Maybe a dupe of #282.
Stacktrace
To Reproduce
Steps to reproduce the behavior:
dasel -f foo.csv --csv-comma ';' -w json --pretty 'equal([], )'
I have to admit I didn't look much at the dasel query syntax (this is a reduced version), I just ended up piping it to
jq
and using someselect
andmap
magic for my usecase :)Expected behavior
It should not crash.
Desktop:
dasel version 2.6.0
The text was updated successfully, but these errors were encountered: