Hidden markov improvements via hmm-lib #49

Closed
wants to merge 39 commits into
from

Projects

None yet

5 participants

@karussell
Member
karussell commented Apr 22, 2016 edited

This change was done by @michaz who integrated GraphHopper with the "hidden markov" work from @stefanholder from the module currently available here: https://github.com/michaz/hmm-lib (minor changes were necessary for the integration)

Thanks a lot to all of them! Please try out and give us feedback!

git clone https://github.com/graphhopper/map-matching.git
cd map-matching
git checkout demoui
git submodule update --init --recursive
mvn package -DskipTests
./map-matching.sh action=import datasource=some.pbf
./map-matching.sh action=start-server
# open browser at localhost:8989 and click browse button to select a local GPX file you want to match

Keep in mind: the GPX track has to be entirely within the area you imported while action=import e.g. use this pbf and the attached GPX files

I also added a simple UI using the matching API.

Several improvements are still necessary and possible, like speed improvements, but the matching itself should already produce better results in most cases and the best: without tweaking any heuristic parameters, just the GPX distance error.

The failing tests needs to be investigated (DONE, mostly parameter tweaking to avoid too precise matching)

The original track is thin and black and the match is greenish:

map-matching-example

michaz and others added some commits Mar 6, 2016
@michaz michaz Seems to do something. 46a0c56
@michaz michaz Use correct distance metric; forward correctly to nearest real node. 634440a
@michaz michaz Relax two tests b5b73fb
@michaz michaz Use correct distance metric; forward correctly to nearest real node. ba8fc13
@michaz michaz Insert checkOrCleanup to make tests succeed. 6ecd04f
@michaz michaz Relax another test. de2ef1e
@michaz michaz Don't "clean up" after all. 1180330
@michaz michaz correct first link b17e189
@michaz michaz .gitmodules f6eecd3
@michaz michaz .gitmodules 17f4392
@michaz michaz Time in seconds instead of milliseconds. 02b6e73
@michaz michaz annex module offline-map-matching 3db58eb
@michaz michaz factored out parameters ad60f60
@michaz michaz factored out parameters c7df871
@michaz michaz Revert simplifying for now. First, get the matching straight. 74e4d3b
@michaz michaz Routes look good, but points are not matched. fe07336
@michaz michaz Finally works. 4bcdcba
@michaz michaz Discovered that a node sequence is not the way to go. We do have a mu…
…ltigraph.
d0d8829
@michaz michaz Use the map. 1d3bc6e
@michaz michaz Use only one querygraph. 62d460e
@michaz michaz Map all virtual edges. 26fddb7
@michaz michaz Fuck. 6d06d88
@michaz michaz Very slow now (the big query graph seems to do it), but works again. …
…Now regenerate the node chain correctly.
eea0d85
@michaz michaz OK, node reconstruction works. c229cc0
@michaz michaz Some cleanup. 357b96c
@karussell karussell merged master ee11a43
@karussell karussell using JS UI instead of vaadin
c91b8de
karussell added some commits Apr 22, 2016
@karussell karussell avoid sys out 73b4aa0
@karussell karussell use bbox for proper zoom
3bf9dc0
@oblonski
Member
oblonski commented Apr 22, 2016 edited

Cool. Thanks. I just followed your instructions more or less blindly and got this:


[ERROR] Failed to execute goal on project map-matching: Could not resolve dependencies for project com.graphhopper:map-matching:jar:0.7-SNAPSHOT: Could not find artifact de.bmw.hmm:hmm:jar:0.2.0-SNAPSHOT in sonatype-oss-public (https://oss.sonatype.org/content/groups/public/) -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
ls: matching-core/target/map-matching-*-dependencies.jar: No such file or directory
Error: Unable to access jarfile action=import

@oblonski
Member

Seems that it cannot find: de.bmw.hmm:hmm:jar:0.2.0-SNAPSHOT
Apart from that the the name looks strange, do I need to install this manually?

@karussell
Member
karussell commented Apr 22, 2016 edited

strange, should build automatically. Can you try:

cd hmm-lib/
mvn clean install

if content in hmm-lib does not exist -> the command git submodule update --init --recursive was somehow not successfully

@oblonski
Member

Ok. This worked. Now it says: Started server at HTTP :8989. :). What is the url I need to get the map matching ui?

@karussell
Member

Edited it: localhost:8989

@oblonski
Member
oblonski commented Apr 22, 2016 edited

I get

{"message": "Not found"}

@karussell
Member
karussell commented Apr 22, 2016 edited

Ah I need to fix the jetty.resourcebase which is ./src/main/webapp. See the next commit

@karussell karussell fix regarding jetty.resourcebase; #49
d0f7b44
@Piskvor
Piskvor commented Jun 15, 2016

After #49 (comment) , I tried starting the server:

$ ./map-matching.sh action=start-server
Exception in thread "main" java.lang.UnsupportedClassVersionError: com/graphhopper/matching/http/MatchServer : Unsupported major.minor version 52.0
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:803)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)

This is on Ubuntu 16.04 with Oracle Java 8; what version 52.0 is unsuitable, and how could I fix this?

@karussell
Member

@Piskvor Thanks for trying this out. What does "java -version" say and what "javac -version"? If 1.7 or lower you need to configure the default java to be at least 1.8

@Piskvor
Piskvor commented Jun 20, 2016

@karussell: Aha! Turns out that javac was correctly set to 1.8, but java was still pointing at the 1.7 install. Thanks!

@karussell
Member

Thanks - good to know - also let us know if you find problems with the new approach. We'll merge this very soon, potentially as optional for now so people can use both, not sure if the old version has advantages though (maybe calculation speed?)

@stefanholder
Collaborator

The hidden Markov approach is certainly slower because it requires to compute the shortest path between all pairs of consecutive map matching candidates. If we assume a constant number n of candidates for each GPS positon then n² shortest paths need to be computed for each GPS position except the first. Instead of computing each shortest path individually, a single-source multi-target routing can be used to speed up performance. This is because only n single-source multi-target routings are necessary per GPS position. It seems that graphhopper already provides a single-source multi-target routing with DijkstraOneToMany.java.

To use single-source multi-target routing with the current hmm-lib API, one needs to compute and save all shortest paths before actually invoking the hmm-lib. The current implementation in MapMatching.java already stores all shortest paths so that the entire matched route can be retrieved later.

I am currently working on an improved hmm-lib API, which can be called iteratively for each time step. This allows calling the single-source multi-target router while the GPS trace is processed. Moreover, the paths between matched GPS positions (the entire matched route) can be retrieved from the hmm-lib after the complete GPS trace is processed, so the caller will no longer need to store all shortest paths. As these improvements still need some time, I suggest using the current hmm-lib API for now and migrating later to the improved version.

Another performance improvement would be to abort the routing as soon as the shortest path would be longer than what could reasonably be travelled in the time between the two GPS positions. In this case the transition probability needs to be set to zero. This is also described in the paper “Hidden Markov Map Matching Through Noise and Sparseness by Paul Newson and John Krumm”, Chapter 4.2.

@karussell
Member
karussell commented Jun 27, 2016 edited

Thanks, sounds good!

It seems that graphhopper already provides a single-source multi-target routing with DijkstraOneToMany.java.

We have a version in the map matcher implemented which is more lightweight per request. See CustomDijkstra

I am currently working on an improved hmm-lib API, which can be called iteratively for each time step.

I'll hopefully have a bit time this week to finally merge this PR. Maybe I keep both algorithms if not too much work.

as soon as the shortest path would be longer than what could reasonably be travelled in the time between the two GPS positions.

I think this is not that simple but we should try it :)

karussell added some commits Jun 28, 2016
@karussell karussell merged master 92e78a8
@karussell karussell include hmm-lib: remove gitsubmodul 7e0f464
@karussell karussell removed vaadin module f228b46
@karussell karussell moved hmmlib to lib2 60f9148
@karussell karussell include hmm-lib directly 8b3d979
@karussell karussell make tests passing 6aa27c1
@karussell karussell use bidir algo instead one source Dijkstra 25fc428
@karussell karussell make web service configurable with gps_accuracy 54c2307
@karussell karussell minor improvements to logs and UI 407bc28
@karussell karussell added a commit that referenced this pull request Jul 1, 2016
@karussell karussell improved matching algorithm via hidden markov, comes with simple UI, #49
652931d
@karussell
Member
karussell commented Jul 1, 2016 edited

After several tests I've merged this. We'll work on performance and quality etc in the master. Currently the old algorithm is completely removed.

Thanks @michaz & @stefanholder !

@karussell karussell closed this Jul 1, 2016
@karussell karussell added this to the 0.8 milestone Jul 1, 2016
@karussell karussell referenced this pull request in bmwcarit/hmm-lib Jul 1, 2016
Closed

official Maven release #4

@karussell karussell changed the title from Hidden markov improvements including JS Demo UI to Hidden markov improvements via hmmlib Jul 1, 2016
@karussell karussell changed the title from Hidden markov improvements via hmmlib to Hidden markov improvements via hmm-lib Jul 1, 2016
@karussell
Member

@harregui saw you do a comparison of map matching for your PhD - would love to get feedback on this one here ... maybe @stefanholder too :)

@stefanholder
Collaborator

Sure, this would be great!

@michaz
michaz commented Jul 13, 2016

Don't read my commit messages, please.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment