Segfault when applying NodeLocationsForWays handler

[ltog]

The following program creates a Segfault when applied to Geofabrik's Switzerland PBF file ( http://download.geofabrik.de/europe/switzerland-latest.osm.pbf ).

#include <osmium/index/map/dummy.hpp>
#include <osmium/index/map/sparse_table.hpp>
#include <osmium/index/map/stl_map.hpp>
#include <osmium/index/map/mmap_vector_anon.hpp>
#include <osmium/handler/node_locations_for_ways.hpp>
#include <osmium/visitor.hpp>
#include <osmium/io/any_input.hpp>
#include <osmium/handler.hpp>
#include <geos/util/IllegalArgumentException.h>
#include <geos/util/GEOSException.h>
#include <osmium/io/any_input.hpp>
#include <osmium/handler.hpp>

typedef osmium::index::map::Dummy<osmium::unsigned_object_id_type, osmium::Location> index_neg_type;
typedef osmium::index::map::SparseTable<osmium::unsigned_object_id_type, osmium::Location> index_pos_type;

typedef osmium::handler::NodeLocationsForWays<index_pos_type, index_neg_type> location_handler_type;


class MyHandler : public osmium::handler::Handler {

public:

    MyHandler() :
        osmium::handler::Handler() {
    }

}; // class MyHandler

int main(int argc, char* argv[]) {

    if (argc != 2) {
        std::cerr << "Usage: " << argv[0] << " OSMFILE\n";
        exit(1);
    }

    MyHandler handler;
    osmium::io::Reader reader(argv[1]);

    index_pos_type index_pos;
    index_neg_type index_neg;
    location_handler_type location_handler(index_pos, index_neg);

    osmium::apply(reader, location_handler, handler);

    return 0;
}

Makefile:

#CXXFLAGS += -O3
CXXFLAGS += -g
CXXFLAGS += -std=c++11 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64

OS:=$(shell uname -s)
ifeq ($(OS),Darwin)
    CXXFLAGS += -stdlib=libc++
    LDFLAGS += -stdlib=libc++
endif

CXXFLAGS_WARNINGS := -Wall -Wextra -pedantic -Wredundant-decls -Wdisabled-optimization -Wctor-dtor-privacy -Wnon-virtual-dtor -Woverloaded-virtual -Wsign-promo -Wold-style-cast

LIB_EXPAT := -lexpat
LIB_PBF   := -pthread -lz -lprotobuf-lite -losmpbf
LIB_BZIP2 := -lbz2


PROGRAMS := segfault

.PHONY: all clean

all: $(PROGRAMS)

segfault: main.cpp
    $(CXX) $(CXXFLAGS) $(CXXFLAGS_WARNINGS) -o $@ $< $(LDFLAGS) $(LIB_EXPAT) $(LIB_PBF) $(LIB_BZIP2)

clean:
    rm -f *.o core $(PROGRAMS)

gdb Output:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff675a700 (LWP 10911)]
[New Thread 0x7ffff5dd3700 (LWP 10912)]
[New Thread 0x7ffff55d2700 (LWP 10913)]
[New Thread 0x7ffff4dd1700 (LWP 10914)]

Program received signal SIGSEGV, Segmentation fault.
0x0000000000434deb in google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> >::sparsegroup (this=0x2981acb0, x=...)
    at /usr/include/google/sparsetable:795
795   sparsegroup(const sparsegroup& x) : group(0), settings(x.settings) {
(gdb) bt
#0  0x0000000000434deb in google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> >::sparsegroup (this=0x2981acb0, 
    x=...) at /usr/include/google/sparsetable:795
#1  0x000000000043ac91 in google::libc_allocator_with_realloc<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > >::construct (this=0x7fffffffdcb0, p=0x2981acb0, val=...) at /usr/include/google/sparsehash/libc_allocator_with_realloc.h:78
#2  0x00000000004392a6 in std::allocator_traits<google::libc_allocator_with_realloc<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > > >::_S_construct<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> ><google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > const&> >(google::libc_allocator_with_realloc<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > >&, std::allocator_traits<google::libc_allocator_with_realloc<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > > >::__construct_helper*, (google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> ><google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > const&>&&)...) (__a=..., 
    __p=0x2981acb0) at /usr/include/c++/4.8/bits/alloc_traits.h:254
#3  0x0000000000437717 in std::allocator_traits<google::libc_allocator_with_realloc<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > > >::construct<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> ><google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > const&> >(google::libc_allocator_with_realloc<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > >&, google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> ><google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > const&>*, (google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> ><google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > const&>&&)...) (__a=..., __p=0x2981acb0) at /usr/include/c++/4.8/bits/alloc_traits.h:393
#4  0x0000000000435031 in std::__uninitialized_fill_n_a<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> >*, unsigned long, google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> >, google::libc_allocator_with_realloc<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > > > (__first=0x2981acb0, __n=669, __x=..., __alloc=...)
    at /usr/include/c++/4.8/bits/stl_uninitialized.h:320
#5  0x0000000000432c1b in std::vector<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> >, google::libc_allocator_with_realloc<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > > >::_M_fill_insert (this=0x7fffffffdcb0, 
    __position=..., __n=669, __x=...) at /usr/include/c++/4.8/bits/vector.tcc:488
#6  0x000000000042f680 in std::vector<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> >, google::libc_allocator_with_realloc<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > > >::insert (this=0x7fffffffdcb0, __position=..., 
    __n=669, __x=...) at /usr/include/c++/4.8/bits/stl_vector.h:1024
#7  0x000000000042afe9 in std::vector<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> >, google::libc_allocator_with_realloc<google::sparsegroup<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> > > >::resize (this=0x7fffffffdcb0, 
    __new_size=43523432, __x=...) at /usr/include/c++/4.8/bits/stl_vector.h:687
#8  0x000000000043948c in google::sparsetable<osmium::Location, (unsigned short)48, google::libc_allocator_with_realloc<osmium::Location> >::resize (this=0x7fffffffdcb0, 
    new_size=2089124703) at /usr/include/google/sparsetable:1317
#9  0x0000000000437917 in osmium::index::map::SparseTable<unsigned long, osmium::Location>::set (this=0x7fffffffdca0, id=2089114703, value=...)
    at /usr/include/osmium/index/map/sparse_table.hpp:91
#10 0x000000000043535e in osmium::handler::NodeLocationsForWays<osmium::index::map::SparseTable<unsigned long, osmium::Location>, osmium::index::map::Dummy<unsigned long, osmium::Location> >::node (this=0x7fffffffdc80, node=...) at /usr/include/osmium/handler/node_locations_for_ways.hpp:102
---Type <return> to continue, or q <return> to quit---
#11 0x000000000043317c in osmium::visitor::detail::switch_on_type<osmium::handler::NodeLocationsForWays<osmium::index::map::SparseTable<unsigned long, osmium::Location>, osmium::index::map::Dummy<unsigned long, osmium::Location> >, osmium::memory::Item> (visitor=..., item=...) at /usr/include/osmium/visitor.hpp:101
#12 0x000000000042f963 in osmium::visitor::detail::apply_item_recurse<osmium::handler::NodeLocationsForWays<osmium::index::map::SparseTable<unsigned long, osmium::Location>, osmium::index::map::Dummy<unsigned long, osmium::Location> >, osmium::memory::Item> (item=..., visitor=...) at /usr/include/osmium/visitor.hpp:153
#13 0x000000000042b577 in osmium::visitor::detail::apply_item_recurse<osmium::handler::NodeLocationsForWays<osmium::index::map::SparseTable<unsigned long, osmium::Location>, osmium::index::map::Dummy<unsigned long, osmium::Location> >, osmium::memory::Item, MyHandler> (item=..., visitor=...) at /usr/include/osmium/visitor.hpp:158
#14 0x0000000000425309 in osmium::apply<osmium::io::InputIterator<osmium::io::Reader, osmium::memory::Item>, osmium::handler::NodeLocationsForWays<osmium::index::map::SparseTable<unsigned long, osmium::Location>, osmium::index::map::Dummy<unsigned long, osmium::Location> >, MyHandler> (it=..., end=...) at /usr/include/osmium/visitor.hpp:242
#15 0x000000000041ea52 in osmium::apply<osmium::io::Reader, osmium::handler::NodeLocationsForWays<osmium::index::map::SparseTable<unsigned long, osmium::Location>, osmium::index::map::Dummy<unsigned long, osmium::Location> >, MyHandler> (source=...) at /usr/include/osmium/visitor.hpp:249
#16 0x00000000004069d6 in main (argc=2, argv=0x7fffffffdf18) at main.cpp:45
(gdb) 

[joto]

The program works for me. What machine are you using? SparseTable doesn't work on 32bit machines, but it should fail when compiling on 32bit.

[ltog]

I'm using 64-bit Ubuntu. Interestingly, using the attached code the segfault happens only in a VM (on VMware Player 6.0.1) running a quite out-of-the-box installation of Ubuntu 13.10 Desktop 64-bit. When I copy the binary to the host system (Ubuntu 64-bit as well), the segfault doesn't happen. (I encountered a segfault on the host system though using another, more complex program and the Germany extract.)

The Andorra file ( http://download.geofabrik.de/europe/andorra-latest.osm.pbf ) seems to be affected as well. It is much smaller than the Switzerland file and may therefore be more convenient to use.

What can I do to shed more light on this issue?

[joto]

This is probably just an out-of-memory condition. See joto/osmium#70 for the same issue with the old Osmium. Unfortunately the Sparsetable implementation doesn't tell us when it is running out of memory, so I can't detect that. Looking at the Sparsetable code there are a few checks and asserts, but not everywhere where it might do some memory allocation.

The Sparsetable development seems to have stopped, the code abandonned. Maybe we have to switch to a different default index for our examples.

You can try using osmium::index::map::VectorBasedSparseMap instead of osmium::index::map::SparseTable. It is also quite space efficient for small datasets.

[ltog]

It seems, it was indeed a problem of having not enough memory. I didn't recognize it, because the System Monitor still showed a lot of free memory. Adding swap space fixed the issue.