How to help with public transport routing?

[tannishpage]

All the bus, ferry and train stations are already on the map in my city, with the routes lined out. But I can't access the routing feature, it says "Subway navigation in this region is not available yet". So, what can I do to help out?

[biodranik]

You may start from here: https://github.com/organicmaps/organicmaps/blob/master/docs/EXPERIMENTAL_PUBLIC_TRANSPORT_SUPPORT.md

It would be great if more people from the community help us with public transport. It's not an easy task.

There is also a great guide from OSMand how to contribute Public Transport info into OpenStreetMap: https://osmand.net/blog/guideline-pt

[csidirop]

Is there a reason why you just added "subway, light rail, monorail and train routes and stops" and no tram or bus routes?

[biodranik]

Other developers started to implement the public transport feature (subways) and did not finish it yet. We would gladly finish it with the help of the community.

[csidirop]

I get an ModuleNotFound Error for requests. Where do the compiler found that import?

File "C:\osmtest\download_gtfs.py", line 15, in
import requests
ModuleNotFoundError: No module named 'requests'

Edit: i downloaded all python scripts, but there are far too many for me to find the missing class. Any idea?

[biodranik]

https://docs.python-requests.org/en/master/user/install/#install

[csidirop]

Haha, here you can see my lack of python knowledge. But it seams that i have an other issue, because requests is already installed.
I'll come back when i've solved that issue

[csidirop]

Fixed that other stupid issue. But the script won't work. I used the way without the omd key, mostly because their site is broken and i was not able to login. Here the log:

ERROR:main:HTTP error 401 Client Error: Unauthorized for url: https://api.transit.land/api/v1/feeds/?per_page=50 downloading zip from https://api.transit.land/api/v1/feeds/?per_page=50
ERROR:main:HTTP error 401 Client Error: Unauthorized for url: https://api.transit.land/api/v1/feeds/?per_page=50 downloading zip from https://api.transit.land/api/v1/feeds/?per_page=50
ERROR:main:Empty urls from C:\osmtest

Looking at https://www.transit.land/ it seams that they need an API key too:

Transitland developers: As of June 9, 2021 developers need to include an API key on all requests to the Transitland v1 Datastore API. This will help us to better manage use and ensure all can access Transitland resources with fewer interruptions. Sign-up is free.

[tannishpage]

@csidirop Have you been able to make it work with a key? Or does the download_gtfs script need modification for it to work?

[csidirop]

@tannishpage no, i took a quick look at the pythonfile and didn't found any parameter for a transitland key (corrct me if i missed something). And since this requirement is new, and the file 6 months old, Im sure it needs modification.

And because this exceeds my nearly non-existent python knowledge, I got nowhere. I could have a look at it this weekend if I find the time, if one of the devs hasn't taken it over by then.

[tannishpage]

@csidirop I'll have a go at it. I've got some experience with python

[csidirop]

@tannishpage great! Let me know if there is something I can do

[tannishpage]

@csidirop I made the modifications to the file and it is working now. It is downloading all the gtfs data from transitland. However, some urls on transitland are causing 403, but that's not really the scripts fault. I'll make a pull request once it finishes running and I'm sure it's not causing any issues.

[tannishpage]

I am trying to compile the gtfs_converter.cpp file using cmake. But I keep getting this error. Am I doing something wrong? Are there any instructions on how to compile this?

CMake Error at CMakeLists.txt:5 (omim_add_executable): Unknown CMake command "omim_add_executable".

[biodranik]

Try to build gtfs_converter using CMakeLists.txt at the root of the project.

[tannishpage]

@biodranik How would I do that. I've never used cmake before. cmake --install CMakeLists.txt This is what I was running before. cmake --install CMakeLists.txt transit/world_feed/gtfs_converter I tried this and got the same error.

[csidirop]

@tannishpage updated download_gtfs.py works great under Win

[csidirop]

Btw, for everyone else trying that: Download can take over over an hour and needs more than 35GB.
But like tannishpage Im stuck with converting the gtfs.

[biodranik]

tools/unix/build_omim.sh -r world_feed command should build gtfs_converter

[tannishpage]

So I did ./configure, because tools/unix/build_omim.sh -r world_feed asked me to do that.

When I run tools/unix/build_omim.sh -r world_feed, it gives me the following output

CMake Warning at CMakeLists.txt:201 (find_package):
  By not providing "FindQt5Core.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "Qt5Core", but
  CMake did not find one.

  Could not find a package configuration file provided by "Qt5Core" with any
  of the following names:

    Qt5CoreConfig.cmake
    qt5core-config.cmake

  Add the installation prefix of "Qt5Core" to CMAKE_PREFIX_PATH or set
  "Qt5Core_DIR" to a directory containing one of the above files.  If
  "Qt5Core" provides a separate development package or SDK, be sure it has
  been installed.


CMake Error at CMakeLists.txt:203 (message):
  Qt5 cmake files were not found, please set QT_PATH environment variable


-- Configuring incomplete, errors occurred!
See also "/home/tannishpage/Documents/omim-build-release/CMakeFiles/CMakeOutput.log".
See also "/home/tannishpage/Documents/omim-build-release/CMakeFiles/CMakeError.log".

Here is CMakeError.log:

Performing C SOURCE FILE Test CMAKE_HAVE_LIBC_PTHREAD failed with the following output:
Change Dir: /home/tannishpage/Documents/omim-build-release/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/bin/make -f Makefile cmTC_0f32c/fast && /usr/bin/make  -f CMakeFiles/cmTC_0f32c.dir/build.make CMakeFiles/cmTC_0f32c.dir/build
make[1]: Entering directory '/home/tannishpage/Documents/omim-build-release/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_0f32c.dir/src.c.o
/usr/bin/cc -DCMAKE_HAVE_LIBC_PTHREAD  -fPIE -std=c11 -o CMakeFiles/cmTC_0f32c.dir/src.c.o -c /home/tannishpage/Documents/omim-build-release/CMakeFiles/CMakeTmp/src.c
Linking C executable cmTC_0f32c
/snap/cmake/918/bin/cmake -E cmake_link_script CMakeFiles/cmTC_0f32c.dir/link.txt --verbose=1
/usr/bin/cc -fuse-ld=gold  CMakeFiles/cmTC_0f32c.dir/src.c.o -o cmTC_0f32c 
CMakeFiles/cmTC_0f32c.dir/src.c.o:src.c:function main: error: undefined reference to 'pthread_create'
CMakeFiles/cmTC_0f32c.dir/src.c.o:src.c:function main: error: undefined reference to 'pthread_detach'
CMakeFiles/cmTC_0f32c.dir/src.c.o:src.c:function main: error: undefined reference to 'pthread_cancel'
CMakeFiles/cmTC_0f32c.dir/src.c.o:src.c:function main: error: undefined reference to 'pthread_join'
collect2: error: ld returned 1 exit status
make[1]: *** [CMakeFiles/cmTC_0f32c.dir/build.make:99: cmTC_0f32c] Error 1
make[1]: Leaving directory '/home/tannishpage/Documents/omim-build-release/CMakeFiles/CMakeTmp'
make: *** [Makefile:127: cmTC_0f32c/fast] Error 2


Source file was:
#include <pthread.h>

static void* test_func(void* data)
{
  return data;
}

int main(void)
{
  pthread_t thread;
  pthread_create(&thread, NULL, test_func, NULL);
  pthread_detach(thread);
  pthread_cancel(thread);
  pthread_join(thread, NULL);
  pthread_atfork(NULL, NULL, NULL);
  pthread_exit(NULL);

  return 0;
}

Determining if the function pthread_create exists in the pthreads failed with the following output:
Change Dir: /home/tannishpage/Documents/omim-build-release/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/bin/make -f Makefile cmTC_8495f/fast && /usr/bin/make  -f CMakeFiles/cmTC_8495f.dir/build.make CMakeFiles/cmTC_8495f.dir/build
make[1]: Entering directory '/home/tannishpage/Documents/omim-build-release/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_8495f.dir/CheckFunctionExists.c.o
/usr/bin/cc   -DCHECK_FUNCTION_EXISTS=pthread_create -fPIE -std=c11 -o CMakeFiles/cmTC_8495f.dir/CheckFunctionExists.c.o -c /snap/cmake/918/share/cmake-3.21/Modules/CheckFunctionExists.c
Linking C executable cmTC_8495f
/snap/cmake/918/bin/cmake -E cmake_link_script CMakeFiles/cmTC_8495f.dir/link.txt --verbose=1
/usr/bin/cc  -DCHECK_FUNCTION_EXISTS=pthread_create -fuse-ld=gold  CMakeFiles/cmTC_8495f.dir/CheckFunctionExists.c.o -o cmTC_8495f  -lpthreads 
/usr/bin/ld.gold: error: cannot find -lpthreads
CMakeFiles/cmTC_8495f.dir/CheckFunctionExists.c.o:CheckFunctionExists.c:function main: error: undefined reference to 'pthread_create'
collect2: error: ld returned 1 exit status
make[1]: *** [CMakeFiles/cmTC_8495f.dir/build.make:99: cmTC_8495f] Error 1
make[1]: Leaving directory '/home/tannishpage/Documents/omim-build-release/CMakeFiles/CMakeTmp'
make: *** [Makefile:127: cmTC_8495f/fast] Error 2

I'm running Elementary OS 6. Not really sure what to do

[biodranik]

Did you install qt5 dev package?

[tannishpage]

Ok, so I installed qt5 dev packages and everything worked nicely. I've generatred the json files using
./gtfs_converter --path_mapping=mapping.txt --path_mapping_edges=mapping_edges.txt --path_gtfs_feeds=dir_for_storing_feeds --path_json=result_json_dir --path_resources=./data

Step 3 is asking me to find a file called map_generator.ini which I can't seem to find.

[biodranik]

There is a link to the doc in my link above:

https://github.com/organicmaps/organicmaps/blob/master/tools/python/maps_generator/README.md

[Altonss]

The public transport routing seems a little bit weird in a certain city. It just uses the metro, and the tramway is not used despite being the shorter route. Is this due to the map view called "Metro" only containing the metro lines and that is used for routing?

[biodranik]

Trams are not used/coded yet.

[biodranik]

Here is the list of OpenStreetMap metro/subways validation results:

https://cdn.organicmaps.app/subway/
or
https://maps.mail.ru/osm/tools/subways/latest/
or
https://alexey-zakharenkov.github.io/subways/rapid/

You can help us by fixing red metro lines directly in OpenStreetMap. Then every @organicmaps user will have an updated Public Transport map (subways and S-Bahn at the moment).

Some general instructions are here:

https://wiki.openstreetmap.org/wiki/Metro_Mapping

And here is info how to validate a separate city:

https://github.com/alexey-zakharenkov/subways/

[biodranik]

@contrapunctus-1 let's discuss and plan it!

[julianfairfax]

GNOME Maps uses https://transport.opendata.ch/ for Switzerland. Hopefully this'll make its way into Transportr at some point soon (see grote/Transportr#768), but it could also be used here, as it works perfectly fine in GNOME Maps.

[yellowhat]

Hi,
I am trying to run gtfs_converter, this is what I run on Fedora 35:

dnf install -y gcc g++ cmake qt5-qtbase-devel qt5-qtsvg-devel zlib-devel libicu-devel freetype-devel git python python-requests

git clone https://github.com/organicmaps/organicmaps --depth 1
cd organicmaps
git submodule update --init --recursive

./configure.sh
tools/unix/build_omim.sh -r world_feed

But still gtfs_converter is not compiled:

$ find . -iname "gtfs_*"
./transit/world_feed/gtfs_converter
./transit/world_feed/gtfs_converter/gtfs_converter.cpp

Does anyone have an end-to-end script to compile the "full public transport support".

Thanks

[biodranik]

Dis you try to use gtfs_converter target instead?

[yellowhat]

$ dnf install -y gcc g++ cmake qt5-qtbase-devel qt5-qtsvg-devel zlib-devel libicu-devel freetype-devel git python python-requests
$ git clone https://github.com/organicmaps/organicmaps --depth 1
$ cd organicmaps
$ git submodule update --init --recursive
$ cd transit/world_feed/gtfs_converter
$ cmake .
-- The C compiler identification is GNU 11.3.1
-- The CXX compiler identification is GNU 11.3.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at CMakeLists.txt:3 (omim_add_executable):
  Unknown CMake command "omim_add_executable".


CMake Warning (dev) in CMakeLists.txt:
  No cmake_minimum_required command is present.  A line of code such as

    cmake_minimum_required(VERSION 3.22)

  should be added at the top of the file.  The version specified may be lower
  if you wish to support older CMake versions for this project.  For more
  information run "cmake --help-policy CMP0000".
This warning is for project developers.  Use -Wno-dev to suppress it.
$ cd ../../../
$ ./configure.sh
$ tools/unix/build_omim.sh -r world_feed                     
$ cmake transit/world_feed/gtfs_converter
CMake Error at CMakeLists.txt:3 (omim_add_executable):
  Unknown CMake command "omim_add_executable".


CMake Warning (dev) in CMakeLists.txt:
  No cmake_minimum_required command is present.  A line of code such as

    cmake_minimum_required(VERSION 3.22)

  should be added at the top of the file.  The version specified may be lower
  if you wish to support older CMake versions for this project.  For more
  information run "cmake --help-policy CMP0000".
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Configuring incomplete, errors occurred!

[yellowhat]

I am able to compile it using CMAKE_CONFIG="-DUNITY_DISABLE=ON" tools/unix/build_omim.sh -c -r gtfs_converter, the binary is under ../omim-build-release/gtfs_converter (took some time to find out).

While using the download_gtfs.py script seems that this line needs to be changed to if feed["spec"].lower() != "gtfs": as now the value for feed["spec"] is GTFS not gtfs, not sure why.

[gerion0]

Just want to mention KPublicTransport here which is a C++/Qt library from the KDE project that provides (online) public transport routing.

[koenvanhollebeke]

I am not able to execute the gtfs_converter: getting stuck by how to specify the file:///path/to/file for the dir_for_storing_feeds, result_json_dir & path_resources. See below.

./gtfs_converter --path_mapping=mapping.txt --path_mapping_edges=mapping_edges.txt --path_gtfs_feeds=dir_for_storing_feeds --path_json=result_json_dir --path_resources=./data
LOG TID(1) WARNING 2.484e-05 gtfs_converter/gtfs_converter.cpp:295 main(): Some paths set in options are not valid. Check the directories: dir_for_storing_feeds result_json_dir ./data
gtfs_converter: Reads GTFS feeds or subway transit.json, produces json with global ids for generator.

Executing the gtfs_converter from ~/omim-build-release$ in Ubuntu 22.04. How to define those locations? "file:///path/to/file" doesn't work either:

./gtfs_converter --path_mapping=mapping.txt --path_mapping_edges=mapping_edges.txt --path_gtfs_feeds="file:///home/koenvh/Downloads/" --path_json="file:///home/koenvh/Downloads/" --path_resources="file:///home/koenvh/omim-build-release/data/"
LOG TID(1) WARNING 3.4137e-05 gtfs_converter/gtfs_converter.cpp:295 main(): Some paths set in options are not valid. Check the directories: file:///home/koenvh/Downloads/ file:///home/koenvh/Downloads/ file:///home/koenvh/omim-build-release/data/
gtfs_converter: Reads GTFS feeds or subway transit.json, produces json with global ids for generator.

[koenvanhollebeke]

I am not able to execute the gtfs_converter: getting stuck by how to specify the file:///path/to/file for the dir_for_storing_feeds, result_json_dir & path_resources. See below.

./gtfs_converter --path_mapping=mapping.txt --path_mapping_edges=mapping_edges.txt --path_gtfs_feeds=dir_for_storing_feeds --path_json=result_json_dir --path_resources=./data LOG TID(1) WARNING 2.484e-05 gtfs_converter/gtfs_converter.cpp:295 main(): Some paths set in options are not valid. Check the directories: dir_for_storing_feeds result_json_dir ./data gtfs_converter: Reads GTFS feeds or subway transit.json, produces json with global ids for generator.

Executing the gtfs_converter from ~/omim-build-release$ in Ubuntu 22.04. How to define those locations? "file:///path/to/file" doesn't work either:

./gtfs_converter --path_mapping=mapping.txt --path_mapping_edges=mapping_edges.txt --path_gtfs_feeds="file:///home/koenvh/Downloads/" --path_json="file:///home/koenvh/Downloads/" --path_resources="file:///home/koenvh/omim-build-release/data/" LOG TID(1) WARNING 3.4137e-05 gtfs_converter/gtfs_converter.cpp:295 main(): Some paths set in options are not valid. Check the directories: file:///home/koenvh/Downloads/ file:///home/koenvh/Downloads/ file:///home/koenvh/omim-build-release/data/ gtfs_converter: Reads GTFS feeds or subway transit.json, produces json with global ids for generator.

Solved: the correct formatting is
./gtfs_converter --path_mapping=mapping.txt --path_mapping_edges=mapping_edges.txt --path_gtfs_feeds=/home/koenvh/Downloads/ --path_json=/home/koenvh/Downloads/ --path_resources=/home/koenvh/omim-build-release/data/

[Altonss]

Hello,

I love OrganicMaps, and I'm following the question about public transport routing. I don't know if you are aware of it, but there is already an Android app called Transportr which does worldwide public transport routing based on https://github.com/schildbach/public-transport-enabler.

The app development has been slowing down last years, but an ongoing effort is trying to accelerate development again grote/Transportr#852 !

So what is my point?

I think that since we have a common goal, it might be appropriate to join forces: The goal is to provide efficient, up-to-date, free and privacy-friendly public transport routing! So I know, the technologies used are not the same (PublicTransportEnabler vs GTFS, Android vs Multiplatform, different licenses...). But maybe there could be some joint efforts or reflections!

That's it for my ideas, I look forward to your thoughts :)

[RedAuburn]

I think that since we have a common goal, it might be appropriate to join forces: The goal is to provide efficient, up-to-date, free and privacy-friendly public transport routing! So I know, the technologies used are not the same (PublicTransportEnabler vs GTFS, Android vs Multiplatform, different licenses...). But maybe there could be some joint efforts or reflections!

That's it for my ideas, I look forward to your thoughts :)

YES!! I was following the Transportr revival, and wishing it could be integrated into OM in some way! 😄
CC @biodranik @vng @rtsisyk this would be great

[gerion0]

public-transport-enabler (is btw also used by and written for Öffi) is written in Java. I would expect that to be a major drawback in integrating it. Maybe some kind of sharing would be possible, so to "export" a route from Transportr to OM but that needs a proper sharing format. The already linked KPublicTransport library is basically an already existing rewrite in C++/Qt (the same technology used by OM).

There also exists an Android/KDE/Qt app KTrip that clones the features of Transportr/Öffi but without the nice GUI (at least the last time I checked that, installing is a bit of a pity, check this out). Anyway, it implements lots of the same APIs for querying public transport services.

[dikkechill]

I see a number of nice suggestions to improve OrganicMaps using other opensource projects.

I do think it's good to distinguish:

1. Offline transport data (e.g. using GTFS)
2. Online transport data
   a. Routing on the device (e.g. using GTFS real time)
   b. Routing using an online services (e.g using something like public-transport-enabler)

The main benefit of OrganicMaps is you can use it offline.

As I understand it, the goal is to optimize for offline data first.
Much of the logic is already implemented.
GTFS is already uniformly formatted using an open standard.
One could also use the same (monthly) update interval as with new OSM maps.

Online / real-time updates add more complexity.
Depending on the implementation it additionally it raises other questions as "do you do the routing on the device or use some online service?" (which public-transport-enabler seems to do).
This in turn raises questions for e.g. privacy, connectivity issues, etc.
Hence I would only look at this only in a second stage and then to use GTFS real time data as 'delta updates' to update changed schedules.

In short, I think the path of least effort and maximum impact is to further implement and test with GTFS data.
Maybe there are other thoughts though.

[dikkechill]

@biodranik
Would it be an idea to create a list of specific tasks - as in other 'master issue tickets e.g. for navigation' - that need to be done by people from the community?

• Which parts needs to be tested exactly?
• What still needs to be added to make public transport work generally in OrganicMaps (e.g. viewing works, but routing doesn't?)
• Which known bugs still need to be fixed?

[biodranik]

Creating a meta issue that links to subtasks is a good idea.

Our experience and current vision of the architecture is:

• Server collects schedules from different sources. Efficient implementation in golang, rust or C++ is preferred for easier scaling and maintenance.
• Different formats are converted to the universal one that can be read by Organic Maps and other clients to allow the following functionality:
• Clients are periodically updating schedule info, that is cached on them and can be used offline, if there is no connection. With the connection real-time info is available.
• Clients build routes from this schedule info offline.
• To scale efficiently, this format should be compact and easily cacheable on HTTP CDN.

There are many subtasks here, feel free to go deeper.

The main goal is to make a worldwide public transport routing better than Google, that is open-source and available to everyone. Development and support can be sponsored by cities and countries, by funds, and by the community.

[dikkechill]

Creating a meta issue that links to subtasks is a good idea.

Added a meta issue #5331 to collect all subtasks to implement public transport routing, please feel free to add

[biodranik]

Closing this issue in favor of #5331