Refactor nodes file #4036

oxidase · 2017-05-12T11:10:51Z

Issue

PR fixes #4028

Tasklist

Cleanup fileIndex, nodes and nodes_data files
~~[ ] Use Geometry views to prevent vector copies~~
Check size reduction for germany-latest.osm.pbf
update relevant Wiki pages
review
adjust for comments

Requirements / Relations

PR partially duplicates #3446

oxidase · 2017-05-12T15:27:25Z

Changed sizes:

germany-latest.osrm.fileIndex 817152000 -> 614072320 ~25%
germany-latest.osrm.ramIndex 836852-> 629372 ~25%
germany-latest.osrm.names 9595145->9594817 = -328 must be checked

oxidase · 2017-05-15T22:14:24Z

@TheMarex u and v can also be removed, but it will require indirection forward_segment_id -> geometry_id -> geometry -> geometry[fwd_segment_position] -> coordinates. So phantom nodes snapping will require a copy of a geometry vector. The copying of vectors can be avoided by returning boost::iterator_range but this will expose internal iterator types in a datafacade, so i would like to skip it in this PR. Also removing reverse_segment_id will require an additional index and/or changes in snapping phantom nodes.

With the current implementation size of osrm::extractor::EdgeBasedNode changes from 32 to 20 bytes.

oxidase · 2017-05-16T08:04:49Z

Size reduction for modified files of germany-latest.osrm is 26%:

fileIndex 817152000->511221760 -37%
ramIndex 836852->524036 -37%
nodes_data 126112613->182162657 +44%

EDIT: single-threaded extraction results for the branch and master

-rw-r--r-- 1 miha miha  9594992 May 16 11:29 germany-latest.osrm.names
-rw-r--r-- 1 miha miha  9594992 May 16 10:36 germany-latest.osrm.names.master

oxidase · 2017-05-16T12:37:33Z

Runtime memory consumption for CH+MLD germany-latest.osm.pbf data slightly increased from

[info] Allocating shared memory of 3731803922 bytes

to

[info] Allocating shared memory of 3787552858 bytes

because part of fileIndex data moved into a shared memory block

TheMarex · 2017-05-16T15:11:56Z

@oxidase is that a measurement against master or against 5.7.0?

oxidase · 2017-05-16T15:55:28Z

@TheMarex no, it is master against the branch

TheMarex

Looks good, just some smaller naming related things. 👍

TheMarex · 2017-05-16T16:05:44Z

include/extractor/edge_based_graph_factory.hpp

@@ -93,7 +93,8 @@ class EdgeBasedGraphFactory

    // The following get access functions destroy the content in the factory
    void GetEdgeBasedEdges(util::DeallocatingVector<EdgeBasedEdge> &edges);
-    void GetEdgeBasedNodes(std::vector<EdgeBasedNode> &nodes);
+    void GetEdgeBasedNodes(EdgeBasedNodeDataContainer &ebg_node_data_container);
+    void GetNodeBasedEdges(std::vector<EdgeBasedNode> &nodes);


The naming here is a little bit confusing. This function returns a EdgeBasedNode but is named GetNodeBasedEdge. I think we should rename that class, it does not represent a single EdgeBasedNode but actually a single segment. Maybe ExternalSegmentData?

TheMarex · 2017-05-16T16:09:18Z

include/extractor/node_data_container.hpp

@@ -65,6 +68,13 @@ template <storage::Ownership Ownership> class EdgeBasedNodeDataContainerImpl
        travel_modes[node_id] = travel_mode;
    }

+    // Used by EdgeBasedGraphFactory to fill data structure
+    template <typename = std::enable_if<Ownership == storage::Ownership::Container>>
+    void SetData(NodeID node_id, ComponentID component_id)


Maybe a less generic name: SetComponentID.

oxidase · 2017-05-16T17:38:03Z

src/extractor/extractor.cpp

@@ -359,18 +362,6 @@ void Extractor::FindComponents(unsigned max_edge_id,
        }
    }

-    // connect forward and backward nodes of each edge


@TheMarex just forgot to mention about the removed part that was introduced in c80c223. i don't get idea why graph must be extended with potentially non-exisiting edges. So if there is a use case for it, the place must be reverted before merge

This code is necessary to enforce that the component ID of the forward and backward node is the same (the implicit assumption is that we are always about to do a u-turn on the spot). There are of course cases where this is generally not true, but we can't handle these right now anyway (for that we would need to split the PhantomNode into forward/backward and have two component IDs).

oxidase · 2017-05-16T17:38:35Z

@TheMarex as discussed per chat EdgeBasedNode is renamed to EdgeBasedNodeSegment

TheMarex added the Work In Progress label May 15, 2017

oxidase force-pushed the refactor/nodes-file branch from 0d495bd to 2a3f890 Compare May 15, 2017 09:08

TheMarex mentioned this pull request May 15, 2017

De-duplicate edge-based-node data in RTree #3446

Closed

3 tasks

TheMarex added this to the 5.8.0 milestone May 15, 2017

oxidase force-pushed the refactor/nodes-file branch 2 times, most recently from 3969e88 to e699a4a Compare May 15, 2017 20:37

oxidase added Refactor Review and removed Work In Progress labels May 16, 2017

TheMarex approved these changes May 16, 2017

View reviewed changes

TheMarex added Review - In feedback and removed Review labels May 16, 2017

oxidase added 6 commits May 16, 2017 18:29

Merge part of .nodes fields into .nodes_data

8dba0bb

Remove data duplicates in .node file

32a92ad

Initialize unused bits in PhantomNode

2b11581

Move ComponentID to EdgeBasedNodeDataContainer

bd22445

Change EdgeBasedNodeDataExternalContainer to EdgeBasedNodeDataContainer

afcacf5

Rename .nodes -> .nbg_nodes and .nodes_data -> .ebg_nodes

ecbeee6

oxidase force-pushed the refactor/nodes-file branch from b76ba89 to 5bfb104 Compare May 16, 2017 17:23

oxidase commented May 16, 2017

View reviewed changes

oxidase force-pushed the refactor/nodes-file branch from 5bfb104 to 75e6808 Compare May 16, 2017 18:30

Renamed EdgeBasedNode to EdgeBasedNodeSegment

9609565

oxidase force-pushed the refactor/nodes-file branch from 75e6808 to 7a26333 Compare May 17, 2017 05:57

oxidase added 2 commits May 17, 2017 09:14

Reverted connection of forward and backward nodes in one SCC

cabaf60

Added ChangeLog entry

3ee629f

oxidase force-pushed the refactor/nodes-file branch from 7a26333 to 3ee629f Compare May 17, 2017 07:15

oxidase added Ready To Merge and removed Review - In feedback labels May 17, 2017

TheMarex merged commit 3546abc into master May 17, 2017

TheMarex deleted the refactor/nodes-file branch May 17, 2017 15:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor nodes file #4036

Refactor nodes file #4036

oxidase commented May 12, 2017 •

edited by TheMarex

oxidase commented May 12, 2017 •

edited

oxidase commented May 15, 2017

oxidase commented May 16, 2017 •

edited

oxidase commented May 16, 2017

TheMarex commented May 16, 2017

oxidase commented May 16, 2017

TheMarex left a comment

TheMarex May 16, 2017

TheMarex May 16, 2017

oxidase May 16, 2017 •

edited

TheMarex May 16, 2017

oxidase commented May 16, 2017

Refactor nodes file #4036

Refactor nodes file #4036

Conversation

oxidase commented May 12, 2017 • edited by TheMarex

Issue

Tasklist

Requirements / Relations

oxidase commented May 12, 2017 • edited

oxidase commented May 15, 2017

oxidase commented May 16, 2017 • edited

oxidase commented May 16, 2017

TheMarex commented May 16, 2017

oxidase commented May 16, 2017

TheMarex left a comment

Choose a reason for hiding this comment

TheMarex May 16, 2017

Choose a reason for hiding this comment

TheMarex May 16, 2017

Choose a reason for hiding this comment

oxidase May 16, 2017 • edited

Choose a reason for hiding this comment

TheMarex May 16, 2017

Choose a reason for hiding this comment

oxidase commented May 16, 2017

oxidase commented May 12, 2017 •

edited by TheMarex

oxidase commented May 12, 2017 •

edited

oxidase commented May 16, 2017 •

edited

oxidase May 16, 2017 •

edited