New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Read CSV files from an executable via pipe #2563
Comments
This is already a common usage of Mapnik. The recommended approach is to write a python or node.js (or C++ for that matter) script to download your data and pass it along in-memory to Mapnik. The cheapest (and slowest) way to do this is to template the XML and use the CSV plugin like:
You could modify https://github.com/springmeyer/coalition which uses this approach but switches out a postgis sql query - instead you would switch out a chunk of data + the CSV plugin. A higher performance approach would be to do the same but use GeoJSON + the GeoJSON plugin, again with the After that the highest performance method would be passing your data in as vector tiles. |
Is it possible to integrate this into mod_tile/renderd? |
Yes, it is surely possible but my only involvement in mod_tile/renderd is helping ensure it builds, I don't do any active development on it so you'll need to code it yourself or ask for help from mod_tle developers. |
Hmm, I thought about this, but integrating it into the frontend is not a very practical solution. I'm using Mapnik with a variety of programs, amongst others renderd, node-mapnik and nik2img. So I would need to patch the re-loading of the Mapnik style for all these programs. That's the beauty of the Postgis solution (in https://github.com/plepe/pgmapcss I'm using a database function to pre-process data), as its comprehensive for all these programs. When I'm not using a PostgreSQL/Postgis database anyway (e.g. Overpass API), replacing the database function by an executable would remove dependencies and administrative overhead for users. Do you think it would be a lot of work to change the Mapnik accordingly? It's not urgent, as the Postgis solution works fine for the time being ... |
Currently I'm experimenting with non PostgreSQL based databases like Overpass API. It would be great if Mapnik could render images without having to import a database first - especially for simple overlays. Currently I'm using a (PLPython3U) database function which downloads and preprocesses the data. This seems to work fine, but it still requires an existing database and PLPython3U is a potential security risk.
So I had the idea, if CSV could be read from an executable via pipe. The executable could handle the downloading of data and print the data to be rendered to stdout. Additionally it should be possible to pass bounding box and scale denominator to the executable, e.g. as command line option(s).
This could look like this:
Things that need to be thought of:
According to #919 the CSV file is loaded a parse time. For the described process the "lazy parsing" mode needs to be implemented.
The text was updated successfully, but these errors were encountered: