Permalink
Branch: master
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
127 lines (91 sloc) 5.44 KB

Considerations for full-planet builds

Pelias is designed to work with data ranging from a small city to the entire planet. Small cities do not require particularly significant resources and should be easy. However, full planet builds present many of their own challenges.

Current full planet builds weigh in at just under 600 million documents, and require about 350GB total storage in Elasticsearch.

Fortunately, because of services like AWS and the scalability of Elasticsearch, full planet builds are possible without too much extra effort. The process is no different, it just requires more hardware and takes longer.

The best performance for full planet import comes on machines with fast, local NVMe SSDs, a fast internet connection for downloading data, and many CPUs for paralell processing.

To set expectations, a 36 core machine can complete a Pelias build in about 19 hours.

Recommended processes

Use Docker containers and orchestration

We strongly recommend using Docker to run Pelias. All our services include Dockerfiles and the resulting images are pushed to Docker Hub by our CI. Using these images will drastically reduce the amount of work it takes to set up Pelias and will ensure you are on a known good configuration, minimizing the number of issues you will encounter.

Additionally, there are many great tools for managing container workloads. Simple ones like docker-compose can be used for small installations, and more complex tools like Kubernetes can be great for larger installations. Pelias is extensively tested on both.

Use separate Pelias installations for indexing and production traffic

The requirements for performant and reliable Elasticsearch clusters are very different for importing new data compared to serving queries. It is highly recommended to use one cluster to do imports, save the resulting Elasticsearch index into a snapshot, and then load that snapshot into the cluster used to perform actual geocoding.

Shard count

Shard count is a balance between several factors. In general, higher shard counts allow more parallelism, at the cost of slightly lower efficiency.

The latest guidance from the Elasticsearch team is that shards should be no larger than 50GB, but otherwise having as few shards as possible is best. The most well tested full planet build configuration is to use 12 shards. If you run performance comparisons at different shard counts, be sure to share your findings!

The elasticsearch section of pelias.json can be used to configure the shard count.

{
  "elasticsearch": {
    "settings": {
      "index": {
        "number_of_shards": "5",
      }
    }
  }
}

Force merge your Elasticsearch indices

Pelias Elasticserach indices are generally static, as we do not recommend querying from and importing to an Elasticsearch cluster simultaneously. In such cases, the highest levels of performance can be achieved by force-merging the Elasticsearch index.

Recommended hardware

For a production ready instance of Pelias, capable of supporting a few hundred queries per second across a full planet build, a setup like the following should be sufficient.

Elasticsearch cluster for importing

The main requirement of Elasticsearch is that it has enough disk for a full build. 400GB across the cluster is a good minimum. Increased CPU power is useful to achieve a higher throughput for queries: a full planet build with all importers running in parallel can easily utilize 16 cores ore more.

Elasticsearch cluster for querying

For queries, essentially the only bottleneck is CPU, although more RAM is helpful so Elasticsearch data can be cached. On AWS, c5 instances are significantly more performant than even the c4 instances, and should be used if high performance is needed.

Example configuration: 4 c5.2xlarge (8 CPU, 8GB RAM) to serve 250 RPS

Importer machine

The importers are each single-threaded Node.js processes, which require around 8GB of RAM each with admin lookup enabled. Faster CPUs will help increase the import speed. Running multiple importers in parallel is recommended if the importer machine has enough RAM and CPU to support them.

Example configuration: 1 c5.4xlarge (16 CPU, 32GB RAM), running importers in parallel

Pelias services

Each Pelias service has different memory and CPU requirements. Here are some rough guidelines:

API

RAM: 200MB per instance CPU: Single threaded, one instance can serve at least 500 RPS Disk: None

Placeholder

RAM: 200MB per instance CPU: Single threaded, supports clustering Disk: Requires about 3GB for a full planet index

Libpostal

RAM: 4GB per instance CPU: Multi-threaded, and extremely fast. A single core can serve 8000+ RPS Disk: about 4GB of data storage required

PIP

RAM: ~8GB CPU: 2 cores per instance recommended, which is enough to serve 5000-7000 RPS

Interpolation

RAM: 3GB per instance currently (please follow our efforts to un-bundle libpostal from the interpolation service) CPU: Single core. One instance can serve around 200RPS Disk: 40GB needed for a full planet interpolation dataset