[new page] Shards and Shard Management

[jackzampolin]

Originally: https://github.com/influxdb/influxdb.com/issues/491

Under Concepts:

• how data are organized into shards
  • by time (If the RP has a duration less than 2 days, all shards are 1h shards. If the RP has a duration of 2 days to 6 months, all shards are 1d shards. For durations greater than 6 months, all shards are 7d shards.)
  • by series (modulo assignment)
  • how replicated
  • what duration and expiration means
• SHOW SHARDS
• SHOW SHARD GROUPS, see SHOW SHARD GROUPS influxdb#4792

Probably link to the FEI about data not being dropped after altering a retention policy: https://docs.influxdata.com/influxdb/v0.9/troubleshooting/frequently_encountered_issues/#data-aren-t-dropped-after-altering-a-retention-policy.

[beckettsean]

The current default shard durations are probably fine for most users

Having shards that are too small leads to significant RAM pressure. Each shard on disk has to store its metadata index in memory. Since there is a high degree of series duplication between shards that leads to a lot of duplication in memory for each shard. The query engine also holds iterators with 1000 point buffers for every shard that exists. The query engine needs more memory just to load, let alone query, as the number of shards rises.

This leads us to the conclusion that larger shards are generally better for query throughput as the query engine has fewer things to keep track of and the RAM needs are lower. Also better for startup time, compression and data locality on disk.

Fewer shards would results in less disk usage because the index entries for series keys wouldn't need to be duplicated in every shard. You would also get denser shards with better compression. I don't think CPU or RAM usage would be that different.

One areas where larger shard duration is really needed is with sparse data. Financial/stock tick data seems to run into issues with our defaults. I've talked a few people where they have maybe 1 point per day going back many years split over a few thousands series and they don't intend to expire any data. That ends up creating lots of tiny shards and causes strain in various parts of the system.

It also leads us to the conclusion that having shards that are too large is probably not a big deal except for file transfer operations (backup, restore, compaction, rebalancing). For clusters, it might be better to keep shards small just to make backups and rebalancing more performant.

However, having too many shards will lead to problematic RAM usage. There are few reasons so far to prefer smaller shards, so we generally don't want to recommend shortening shard durations.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.