write multiple PBFs if the protobuf object gets too big #3954
Conversation
src/mjolnir/ingest_transit.cc
Outdated
| std::string get_tile_path(const std::string tile_dir, const GraphId& tile_id) { | ||
| auto file_name = GraphTile::FileSuffix(tile_id); | ||
| file_name = file_name.substr(0, file_name.size() - 3) + "pbf"; | ||
| return tile_dir + filesystem::path::preferred_separator + file_name; | ||
| }; |
There was a problem hiding this comment.
pulled this out into a separate function, it's needed in multiple other functions
|
~~I quickly tried one way of doing it, but I really really don't like it @kevinkreiser . If you have a better idea later, most welcome, otherwise I'll pore over it tmrw to make it nicer. ~~
EDIT: I made it nicer, couldn't stand having it like that. Note, it's failing a test right now, but can only be a small thing |
src/mjolnir/ingest_transit.cc
Outdated
| for (const auto& trip : current.trips) { | ||
| dangles = | ||
| write_stop_pair(tile, current, trip, feeds(trip), platform_node_ids, uniques) || dangles; |
There was a problem hiding this comment.
easier to loop over trips right here..
…till passes even though it's configured with only 1 trip limit, why??
…ate test accordingly
| @@ -33,6 +33,7 @@ config = { | |||
| 'transit_dir': '/data/valhalla/transit', | |||
| 'transit_feeds_dir': '/data/valhalla/transit_feeds', | |||
| 'transit_bounding_box': Optional(str), | |||
| 'transit_pbf_limit': 20000, | |||
There was a problem hiding this comment.
those are representing gtfs trips. let's see what real feeds do. thanks @kevinkreiser for suggesting that.
| lock.unlock(); | ||
| throw std::runtime_error("Couldn't load " + file_name); |
There was a problem hiding this comment.
first unlock, then throw
| if (std::isdigit(fname.back())) { | ||
| // we produce 2 pbf tiles on purpose, where the last one (xx.pbf.0) only has a bunch of stop | ||
| // pairs | ||
| EXPECT_NE(transit.stop_pairs_size(), 0); | ||
| continue; | ||
| } |
There was a problem hiding this comment.
with mjolnir.transit_pbf_limit: 1 we get two PBFs (one with xx.pbf.0), but the last one only has a few stop pairs, most of the objects are in the first one
| * @param file_name name of the file | ||
| * @return transit data of the tile with the given GraphId | ||
| */ | ||
| Transit read_pbf(const baldr::GraphId& id, const std::string& transit_dir, std::string& file_name); |
|
This should be done for now @kevinkreiser .
|
We had a small bug where large PBF objects wouldn't be able to get serialized.