Skip to content

Latest commit

 

History

History
60 lines (43 loc) · 3.93 KB

performance.md

File metadata and controls

60 lines (43 loc) · 3.93 KB
Raw

Performance

Metarank is a secondary re-ranker: it's an extra non-free step in your retrieval process. In a typical scenario, you should expect the following extras:

  • Re-ranking latency: 10-30 ms
  • Redis memory usage: 1-10 GiB
  • Data import throughput: 1000-3000 events/second.

Response latency

On a RankLens dataset in a synthetic latency test, we observed the following distribution:

Response latency depends on request size

Each Metarank installation is unique, but there are common things affecting the overall latency:

  • State encoding format: binary is faster than json due to its compact representation.
  • Metarank-Redis network latency: Metarank pulls all features for all re-ranked items in a single large batch. There are no multiple network calls and only a constant overhead.
  • Request size: the more items you ask to re-rank, the more data needs to be loaded.
  • A number of feature extractors: the more per-item features are defined in the config, the more data is loaded during the request processing.

So while planning your installation, expect Metarank to be within 20-30 ms latency budget.

Memory usage

Using the same reference RankLens dataset, we built a fuzzy synthetic dataset generator and generated the following dataset variations:

  • N users, 100k items.
  • Each user made 2 rankings within a single session.
  • Each ranking event has 2 clicks made by the user.
Users Items Rankings Clicks Total events Uncompressed size
128K 100K 256K 512K 896K 287MiB
256K 100K 512K 1M 1.8M 512MiB
512K 100K 1M 2M 3.5M 963MiB
1M 100K 2M 4M 7.1M 1.8GiB
2M 100K 4M 8M 14.3M 3.5GiB
4M 100K 8M 16M 28.6M 7.1GiB

Metarank only tracks aggregated data required for the re-ranking and does not store raw events. Therefore, memory usage depends on the following characteristics of your dataset:

  • A number of unique users: per-user click-through events are used as input for the ML model training.
  • A number of items: if you define per-item feature extractors (like string or number), current point-in-time values of used fields are persisted.
  • A number of users: if you define per-user features like interacted_with, we persist a per-user list of interacted items.
  • A number of features: stored click-through events also contain snapshots of all per-item feature values used to build the ranking back in time.

The resulting memory usage for a binary state encoding format and Redis as a persistence store is shown in the diagram below:

Redis/heap memory usage

Users, thousands Redis, MB Import time, min
128 386 8
256 448 16
512 561 30
1024 812 61
2048 1210 110
4096 1860 205

So while planning your installation, expect Metarank to use around 0.8 GiB per 1M users.