Kartotherian is a map tile service originally built for the Wikimedia projects. Kartotherian ties together various open source modules from the TileLive ecosystem, thereby providing for serving tiles from a variety of sources.
This version supports Node.js 10+.
- Changelog
- Maps version 2 architecture and infrastructure diagrams
- Maps infrastructure diagrams
- Maps data flow diagrams
- How to create RESTful API services
This code is cross-hosted at gerrit
Maps nodejs server for vector-based tiles and snapshots, designed for Wikipedia and other sites. It ties together a number of MapBox components for vector and raster rendering based on Mapnik 3, and uses service runner for scalability, performance monitoring and stability.
Kartotherian can serve vector and raster tiles in multiple formats and optional scaling:
http://.../{source}/{zoom}/{x}/{y}[@{scale}x].{format}
- The sources are configured with the source config file. Sources configuration supports different methods of tile storage, such as HTTP or files, generation from postgress db, overzoom to extract the tile from lower zooms if missing, layer extraction, mixing multiple sources together, etc.
- Optional scalling can render larger images for high resolution screens (only those enabled in the source, e.g.
[1.5, 2]) - Supported formats include PNG ang JPEG, SVG, PBF vectors, and JSON (with
nogeoandsummarydebug options)
Kartotherian supports static image generation. Users may request a PNG or a JPEG snapshot image of any size, scaling (only those enabled in the source, e.g. [1.5, 2]), and zoom level:
http://.../img/{source},{zoom},{lat},{lon},{width}x{height}[@{scale}x].{format}
# image centered at 42,-3.14, at zoom level 4, size 800x600
http://.../img/osm-intl,4,42,-3.14,800x600.png
# the same but for higher DPI device with 1.5 scaling
http://.../img/osm-intl,4,42,-3.14,800x600@1.5x.png
- Note, that static images that include overlays like markers or geoshapes do only support 2x scaling.
Kartotherian can be used as a source of the PBF data for Mapbox studio. See info about style editing in osm-bright-source. The info data is available at http://.../{style}/pbfinfo.json for pbf source, and http://.../{style}/info.json for the styled image source.
Kartotherian can generate marker images by wrapping any of the maki icons with a pushpin image, in any color. The URL schema is matched to the one used by the mapbox.js.
http://.../v4/marker/pin-l-cafe+de00ff@2x.png
http://.../v4/marker/ {base} - {size:s|m|l} [-{letter-or-digit-or-icon-name}] + {color} [@2x] .png
At this point, only "pin" is supported for the base. The color is a 3 digit or 6 digit hex number. Optional scaling can only be 2x. Beyond the pre-defined maki icons, you may give a number (0-99), a single letter (a-z), or nothing.
git clone https://github.com/kartotherian/kartotherian.git # Clone the repository
cd kartotheriannpm install
node server.js -c config.external.yamlBrowse to http://localhost:6533/
The set up inside sources.external.yaml does not use any storage or caching, so it will not be suitable for production. You will need to to set up your own local database as described in osm-bright.tm2source, which is installed in node_modules/osm-bright-source, and configure additional source chains and setup a proper storage to make this into a production system.
Inside the conf key:
sources- (required) Either a set of subkeys, a filename, or a list of file names. See below on how to configure the sources.variables(optional) - specify a set of variables (string key-value pairs) to be used inside sources, or it could be a filename or a list of filenames/objects.defaultHeaders(optional, object) - a set of extra headers that will be sent to the user unless the source provides its own. (public requests only)headers(optional, object) - a set of extra headers that will be sent to the user instead of the headers returned by the source. (public requests only)
You can also read the documentation.
Kartotherian platform consists of a number of elements, some of which conform to the general specifications established by MapBox, and therefor can reuse components that confirm to the same specification.
core- Loads and configures tile sources, and provides some common utility functions- server - Handles user web requests for tiles and source info, as well as registers additional data type handlers like maki markers and image snapshots.
- maki - Request handler for maki markers, generates PNG marker images that can be used from geojson.
- snapshot - Request handler for static images by combining multiple tiles into one snapshot image of a requested size, with optional geojson overlays based on mapdata stored in MediaWiki.
Tile source to restructure vector PBFs for multilingual usecases, such as convert a single JSON object into multiple key/values, or to replace all language key/value names with a single one.
Tile is generated with 'name_' field set a JSON-encoded key-value object. Babel can be used to convert that tile to a tile, with each value in the object becoming a tag of its own, e.g. 'name_en', 'name_fr', ... . Also, babel can be used to replace multiple 'name_lang' tags with a single 'name_' tag right before rendering it, choosing the best language based on the fallback rules, but only if it is different from the 'name' tag.
# Process tiles from 'gen' source, expanding json string into multiple tags
json2tags:
uri: json2tags://
params:
source: {ref: gen}
tag: name # optional, 'name' is the default# Process tiles from 'store' source, replacing all 'name_*' tags with a single 'name' tag
babel:
uri: babel://
params:
source: {ref: store}
# optional, 'name' is the default
tag: name
# optional, used by default if no 'lang' code is passed to getAsync()
defaultLanguage: 'en'
# optional map of fallback values. Can be a json file or an object value
languageMap: '/my/path/fallback.json'
# -- OR --
languageMap:
en: ['fr', 'es', 'de']
ru: ['be']For babel://, the language of the name_ is chosen based on these rules:
getAsync({z,x,y, lang:'xx'):
name_xx- Use explicitly set fallbacks from the languageMap
- Use any
name_yy-ScriptwhereScriptis the script ofxx. E.g. iflang=ru, pick anylang_yy-Cyrl. - If
xxuses the Latin script, use anyname_zz_rm name
Babel gets the CLDR defined script name (Latn, Cyrl, ... ) based on the language code. It also uses a few overrides from the overrides.json. This file should be updated with any language IDs found in OSM data.
Sources is a way to set up data processing pipelines for Kartotherian. Any source based on tilelive.js specification may be used. Source configuration could be located in standalone files, or be embedded in the main configuration file, or a mix of both. The sources value in the config file could be a string (file), an object defining the source, or an array of strings and objects.
uri is the only mandatory field, and it specifies how tilelive.js
will locate and initialize the new source. The protocol determines which tile provider will be used.
Since sometimes not everything can be added as query parameters to the URI, there is a set of additional keys to help. Values can either be hardcoded as strings/numbers/booleans, or can be calculated on the fly.
A simple source configuration to set up a tile storage as files in the ./vectors dir:
filestore:
uri: file://./vectorsThe path can also be set via a parameter:
filestore:
uri: file://
pathname: ./vectorsThe value does not have to be given in the source, but instead could be dynamically generated.
For example, the env uses an environment variable, and the var generator pulls the value from the variable store.
The variables are defined in a separate file(s), similar to sources.
filestore:
uri: file://
pathname: {env: KARTOTHERIAN_PATH} # Uses an environment variable
# or
pathname: {var: tilepath} # Uses a variable named tilepathMore parameters can be set using params - a set of additional values to be set in URI:
oz:
# "overzoom:" is a tile source that will attempt to get a tile from another source,
# and if tile is missing, attempt to get a portion of the lower-zoom tile.
uri: overzoom://
# Specify the tile source - this adds a properly escaped query parameter
# ?source=sourceref:///?ref=gen
param:
source: {ref: gen}In general, these value substitutions are available:
{env: envVarName}- the value becomes the value of the environment variableenvVarName. This might be useful if you want to make all the settings public except for the passwords that are stored in a secure location.{var: varName}- the value becomes the value of the variablevarNamefrom the variables file / variables conf section of the main config file. This might be useful if you want to make all the settings public except for the passwords that are stored in a secure location.{ref: sourceId}- the value becomes a reference to another source. Some sources function as filters/converters, pulling data internally from other sources and converting the result on the fly. For example, the overzoom source pulls data from another source, and if it's not available, tries to find a lower-zoom tile above the given one, and extract a portion of it. Internally, it uses a forwarding sourceref: source.{npmloader: npm-module-name}or{npmloader: ['npm-module-name', 'arg1', 'arg2', ...]}- if npm module supports loading customization, it should be loaded via the npmloader. Npmloader is only available inside the source'sxmlkey.{npmpath: ['npm-module-name', 'subdir', 'subdir', 'filename']}- some files may be located inside the NPM modules added to the Kartotherian project, i.e. osm-bright-source. To reference a file inside npm, set npm's value to an array, with the first value being the name of the npm module (resolves to the root of the npm module), and all subsequent strings being subdirs and lastly - the name of the file. Subdirs may be omitted.npmpath: ["osm-bright-source", "data.xml"]would resolve to a rooted path/.../node_modules/osm-bright-source/data.xml
The xml parameter is used to load and alter XML for some sources like
tilelive-bridge (SQL→VectorTile or TIFF→RasterTile) and
tilelive-vector (Style VectorTile → PNG).
The xml field must evaluate to the xml file path.
gen: # The name of the source (could be referenced later)
uri: bridge:// # Required - the URI used to construct the source
xml: # Init source with this xml instead of the URI's other parameters
# Set xml to the location of the 'data.xml', which is located inside the osm-bright-source npm
npmpath: ["osm-bright-source", "data.xml"]
xmlSetDataSource: # Before loading, update the datasource section of the standard mapnik config file
if: # Only update datasources that match all these values (logical AND)
dbname: gis # Instead of 'gis', you can use {npmpath:...}, {ref:..}, and {var:...}
host: ''
type: postgis
set: # Replace these keys with the new values
host: localhost
user: {var: osmdb-user} # Instead of hardcoding, use the value from the variables file or conf section
password: {var: osmdb-pswd}xmlSetAttrs- for xml, overrides the attributes of the root element with the new ones. For example, you may change the font directory of the<Map>element:
s2:
uri: vector://
xml:
npmloader: osm-bright-style # stylesheet xml is in npm
xmlSetAttrs:
# Note that this is not needed for osm-bright-style because that module does this internally
font-directory: {npmpath: ["osm-bright-fonts", "fonts/"]}xmlSetParams- for xml, overrides the top level<Parameters>values with the new ones. For example, thevectorsource requires xml stylesheet to point to the proper source of PBFs:
s2:
public: true
uri: vector://
formats: [png,json,headers,svg,jpeg]
xml:
npmloader: osm-bright-style # stylesheet xml is in npm
xmlSetParams:
source: {ref: gen} # set source parameter to the 'gen' sourcexmlLayers- keep all non-layer data, but only keep those layers that are listed in this value (whitelist):
s2:
public: true
uri: vector://
formats: [png,json,headers,svg,jpeg]
xml:
npmloader: osm-bright-style # stylesheet xml is in npm
xmlLayers: ['landuse', 'road'] # Only include these layers when renderingxmlExceptLayers- same asxmlLayers, but instead of whitelisting, blacklist (allow all except these):
s2:
public: true
uri: vector://
formats: [png,json,headers,svg,jpeg]
xml:
npmloader: osm-bright-style # stylesheet xml is in npm
xmlExceptLayers: ['water'] # Exclude water layer when renderingxmlSetDataSource- change all layer's datasources' parameters if they match conditions:ifis a set of parameter values that all must match,xmlLayersandxmlExcludeLayersjust like above set which layers to address, andsetspecifies the new parameter values to be set.
Instead of an object, xmlSetDataSource can be set to an array of objects to provide
multple change sets.
public(boolean) - should this be source be accessible via/<sourceId>/z/x/y.formatrequests. You may also set configuration parameterallSourcesPublicto true to make all sources public (might be dangerous)minzoom(int) - minimum allowable zoom for the public request (public requests only)maxzoom(int) - maximum allowable zoom for the public request (public requests only)defaultHeaders(object) - a set of extra headers that will be sent to the user unless the source provides its own. (public requests only)headers(object) - a set of extra headers that will be sent to the user instead of the headers returned by the source. (public requests only)formats(array of strings) - one string or a list of string values specifying allowed formats, e.g.['png','jpeg']scales(array of numbers) - one number or a list of number values specifying allowed scalings, e.g.[1.3, 1.5, 2, 2.6, 3]setInfo(object) - provide values that will be reported to the client via the/<sourceId>/info.json. See https://github.com/mapbox/tilejson-specoverrideInfo(object) - override values produced by the source's getInfo(), or if value is null, remove it. Result will be accessible via/<sourceId>/info.json. See https://github.com/mapbox/tilejson-spec
Kartotherian service to generate geometric shapes from PostgreSQL data
See https://github.com/kartotherian/kartotherian
To configure, add geoshapes section to the kartotherian configuration with the following parameters:
geoshapes:
host: localhost
database: gis
table: planet_osm_polygon
user: ...
password: ...
maxidcount: (int, optional, default=500) - Maximum number of IDs to allow per request
allowUserQueries: (bool, optional, default=false) - If true, allow sql parameter + args to specify which SQL to use
wikidataQueryService: (string, optional, default=https://query.wikidata.org/bigdata/namespace/wdq/sparql) - Lets user get a list of WikidataIDs from an external Wikidata Query Service. if false, disables.Without this config block, the service will skip its loading
Make sure to create a Postgres index, e.g.:
CREATE INDEX planet_osm_polygon_wikidata
ON planet_osm_polygon ((tags -> 'wikidata'))
WHERE tags ? 'wikidata';Service will return topojson to the queries such as /geoshape?ids=Q1384,Q1166 (get New York and Michigan state shapes).
Save result as a file and upload to http://www.mapshaper.org/ to visualize.
Additionally, the service allows query=... parameter to get the Wikidata IDs from the http://query.wikidata.org service. It calls the service to execute
a query, extracts IDs, and matches them with the shapes in the OSM database. All other values are returned as topojson object properties.
Optional truthy parameter getgeojson=1 will force the result to be returned as geojson rather than topojson.
- kartotherian-overzoom - Tile source that will zoom out if the requested tile does not exist, and extracts the needed portion from the lower-zoom tile it finds.
- osm-bright-source - SQL queries used by the
tilelive-bridgeto generate a vector tile from Postgres Database - osm-bright-style - Style used by the
tilelive-vectorto convert vector tiles into images. - osm-bright-fonts - Fonts used by the
osm-bright-style.
- tilelive - ties together various tile sources, both vector and raster
- tilelive-bridge - generates vector tiles from SQL
- tilelive-vector - converts vector tiles to raster tiles
- abaculus - generates raster images of any location and size from a tile source
- mapnik - Tile rendering library for node
- leaflet - JavaScript library for mobile-friendly interactive maps
This documentation assumes that you are going to use osm-bright.tm2 and osm-bright.tm2source for a map style.
Node.js v6 or v8 required; v10+ currently not supported.
For Windows and Mac, downloadable installers can be found at https://nodejs.org/download/release/latest-v6.x/ or https://nodejs.org/download/release/latest-v8.x/.
For Linux, installation via the instructions at https://nodejs.org/en/download/package-manager/ is recommended.
On Ubuntu these can be installed with
sudo apt-get install git unzip curl build-essential sqlite3 pkg-config libcairo2-dev libjpeg-dev libgif-dev libmapnik-devgit clone https://github.com/kartotherian/kartotherian.git
cd kartotherian
npm installSet up osm-bright.tm2source as described in its documentation..
osm-bright.tm2source is installed in node_modules/osm-bright-source
# 0 - one instance, 1+ - multi-instance with autorestart, ncpu - multi-instance, one per CPU
num_workers: 0
# Host port
port: 6533
# Comment out this line to listen to the web
# interface: localhost
# Place all variables (e.g. passwords) here - either as a filename, or as sub-items.
variables:
# Place all sources you want to serve here - either as a filename, or as sub-items.
# See sources.prod.yaml for examples
sources: sources.yamlUse one of the config files, or update them, and make a link config.yaml to it.
Might require caching headers added to the source/config.
# From https://www.varnish-cache.org/installation/debian
sudo -Hi
apt-get install apt-transport-https
curl https://repo.varnish-cache.org/GPG-key.txt | apt-key add -
echo "deb https://repo.varnish-cache.org/debian/ jessie varnish-4.0" >> /etc/apt/sources.list.d/varnish-cache.list
apt-get update
apt-get install varnish
vi /etc/varnish/default.vclChange default backend to:
backend default {
.host = "localhost";
.port = "6533";
}
Add this to vcl_deliver (to track hits/misses):
if (obj.hits > 0) {
set resp.http.X-Cache = "HIT";
} else {
set resp.http.X-Cache = "MISS";
}
Edit /etc/systemd/system/varnish.service - set proper listening port (80) and cache size:
ExecStart=/usr/sbin/varnishd -a :80 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,4g
In bash:
systemctl daemon-reload # because we changed the .service file
systemctl restart varnish.service
systemctl status varnish.service # check the service started with the right params
varnishstat # monitor varnish performancenpm startIn browser, navigate to http://localhost:6533/.
The docker-start and docker-test scripts are deprecated, and only remain for backwards compatibility. Instead, developers should configure .pipeline/blubber.yaml and install Blubber to generate the desired Dockerfile.
To see the Dockerfile generated by blubber, ensure the blubber CLI is setup and execute:
blubber .pipeline/blubber.yaml {variant}
where variant is one of either build, development, test, etc. in blubber.yaml.
In place of docker-test, to run your service's tests, execute:
blubber .pipeline/blubber.yaml test | docker build --tag service-test --file - .
docker run service-test
In place of docker-start, to run your service, execute:
blubber .pipeline/blubber.yaml production | docker build --tag service-node --file - .
docker run service-node
In a lot of cases when there is an issue with node it helps to recreate the
node_modules directory:
rm -r node_modules
npm installKartotherian emits StatsD metrics, under the following keys:
kartotherian.req.<src>.<zoom>.<format>.static.<scale> - timing metric for successful snapshot request. The parameters correspond to the URL parameters, for example https://maps.wikimedia.org/img/osm-intl,6,53.383333,-1.466667,300x400.png has src="osm-intl", zoom="6", and format="png".
kartotherian.req.<src>.<zoom>.<format>[.<scale>] - timing metric for a tile request.
kartotherian.req.<src>.info - timing metric for info.json request.
kartotherian.marker - timing metric for serving a single Maki marker icon, which will be embedded in an interactive view. Example request https://maps.wikimedia.org/v4/marker/pin-m-museum+a6a6a6.png
kartotherian.<type>.<query> - timing metric for geoshape request. type can be "geoshape", "geoline", or "geopoint". query is either "wdqs" for a SPARQL query, or "ids" for a local postgres lookup.
kartotherian.init - counter incremented when the service starts up.
kartotherian.err.<error> - counter incremented for each error that ends a request. Timing metrics above are not incremented in case of an error.