Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read CSV files from an executable via pipe #2563

Open
plepe opened this issue Nov 5, 2014 · 5 comments
Open

Read CSV files from an executable via pipe #2563

plepe opened this issue Nov 5, 2014 · 5 comments
Labels

Comments

@plepe
Copy link
Contributor

plepe commented Nov 5, 2014

Currently I'm experimenting with non PostgreSQL based databases like Overpass API. It would be great if Mapnik could render images without having to import a database first - especially for simple overlays. Currently I'm using a (PLPython3U) database function which downloads and preprocesses the data. This seems to work fine, but it still requires an existing database and PLPython3U is a potential security risk.
So I had the idea, if CSV could be read from an executable via pipe. The executable could handle the downloading of data and print the data to be rendered to stdout. Additionally it should be possible to pass bounding box and scale denominator to the executable, e.g. as command line option(s).

This could look like this:

<Datasource>
  <Parameter name="type">csv</Parameter>
  <Parameter name="pipe">./download_data.py -b !bbox! -s !scale_denominator!</Parameter>
</Datasource>

Things that need to be thought of:

  • I think Mapnik would need to know the titles of the csv columns at parse time: Maybe the executable could be called with bounding box "init", than the headers without data could be printed.
  • If the executable stops with a non-zero exit code, rendering should be aborted.

According to #919 the CSV file is loaded a parse time. For the described process the "lazy parsing" mode needs to be implemented.

@springmeyer
Copy link
Member

It would be great if Mapnik could render images without having to import a database first

This is already a common usage of Mapnik. The recommended approach is to write a python or node.js (or C++ for that matter) script to download your data and pass it along in-memory to Mapnik. The cheapest (and slowest) way to do this is to template the XML and use the CSV plugin like:

'<Datasource>
  <Parameter name="type">csv</Parameter>
  <Parameter name="inline">'
+ data + 
'</Parameter>
</Datasource>'

You could modify https://github.com/springmeyer/coalition which uses this approach but switches out a postgis sql query - instead you would switch out a chunk of data + the CSV plugin.

A higher performance approach would be to do the same but use GeoJSON + the GeoJSON plugin, again with the inline parameter.

After that the highest performance method would be passing your data in as vector tiles.

@plepe
Copy link
Contributor Author

plepe commented Nov 5, 2014

The cheapest (and slowest) way to do this is to template the XML and use the CSV plugin like:

Is it possible to integrate this into mod_tile/renderd?

@springmeyer
Copy link
Member

Yes, it is surely possible but my only involvement in mod_tile/renderd is helping ensure it builds, I don't do any active development on it so you'll need to code it yourself or ask for help from mod_tle developers.

@plepe
Copy link
Contributor Author

plepe commented Nov 8, 2014

Hmm, I thought about this, but integrating it into the frontend is not a very practical solution. I'm using Mapnik with a variety of programs, amongst others renderd, node-mapnik and nik2img. So I would need to patch the re-loading of the Mapnik style for all these programs. That's the beauty of the Postgis solution (in https://github.com/plepe/pgmapcss I'm using a database function to pre-process data), as its comprehensive for all these programs.

When I'm not using a PostgreSQL/Postgis database anyway (e.g. Overpass API), replacing the database function by an executable would remove dependencies and administrative overhead for users.

Do you think it would be a lot of work to change the Mapnik accordingly? It's not urgent, as the Postgis solution works fine for the time being ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants