Segmentation fault #840
Comments
|
Oh, I see I'm a couple of versions behind. Trying again with 1.34.6 |
|
Slightly different output but still crashed: |
|
@stevage thanks for reporting and helping narrow this down. Getting a backtrace out of the core dump would be ideal, since it should show where the crash is happening. That way we might be able to fix without replicating. https://jvns.ca/blog/2018/04/28/debugging-a-segfault-on-linux/ looks good, but skip the |
|
Ah, great. Ok, I'm trying the logbt option. |
|
Ok, I have emailed you the dump. |
|
Ok, finally got there. With this command line the crash did not occur:
|
|
Thanks @stevage - this is the key line that appears to be causing the segfault:
@ericfischer any idea how to fix this? |
|
I'd also be interested to hear any possible workarounds. |
|
Thanks. This must be indexing off the end of the buffer that encodes a variable-length number, either because it was encoded wrong in the first place, or because the metadata pointer is pointing to the wrong place. I wish I had a workaround to offer immediately, but it looks like it's not as easy as I thought to substitute a different representation for serialized numbers. I'll keep investigating. |
|
The workaround seems to be to insert Line 654 in ddb7993 but I'm still not sure what is causing the externalized metadata to be misencoded. |
|
Is there anything more I can do to help? |
|
OIC. I'll try patching that and see if it prevents the crash. Sorry for the dumb question, but will there be side-effects from that change? I'm not really sure what "externalized metadata" is. |
|
Sorry to be unclear. The "separate metadata" file is basically a holdover from an older version of Tippecanoe in which most of the characteristics of a feature were stored in a separate file so they wouldn't have to be duplicated across temporary files when a feature was clipped into many sub-features in different tiles. I gradually moved more and more things into the "geometry" file, so all it has in it now is a table of indices of key/value attribute pairs into the attribute pool, and is only populated when a feature is expected to span a large number of tiles. I should probably just get rid of it, since it doesn't seem to work reliably and is another thing that has to be handled as a special case when it exists. |
|
Could you please try #845 and see if it fixes the problem for you? |
|
It crashed as before: I forgot to use logbt, so will do that now. |
|
Hmm, so much for that idea. Any chance you could share your data file so I can try to reproduce the exact circumstances? |
|
Trying to find a way to do that. It's 81GB uncompressed, and also subject to licensing rules which I'm trying to get clarity on. In the meantime, here's logbt: |
|
@ericfischer Were you able to make any progress with the file I sent back in Feb? I emailed it to the address on your Github profile page. |
|
@stevage I am facing the exact problem. Did you get a solution over the past year and half now? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I have been processing a file which is 81GB of newline-delimited JSON. I have processed it a couple of times successfully on this machine. Just now I changed the parameters slightly:
and got a seg fault:
Now, following this I'll try running it again to actually get the core dump, which seems to not be written by default.
The changes between "worked" and "crashed" are:
Somewhere in the log I also noticed there was one tile that tippecanoe couldn't make small enough. What does it actually do in that situation? Previously I thought I had seen it abort altogether.
I have also managed to process a subset of this dataset successfully, with these exact settings.
I'm not sure how to share the source file with you, given its size. Any ideas?
tippecanoe v1.32.12The text was updated successfully, but these errors were encountered: