Allow separate retention periods for states and short-term statistics

[hpuac]

Describe your core improvement

Home Assistant Recorder should allow the retention period of short-term statistics to be configured independently from the retention period of regular state history.

Currently, purge_keep_days effectively couples the retention of different types of recorder data. At the same time, recorder.purge_entities allows users to implement more granular retention policies for state history, but does not apply the same retention to statistics_short_term.

A configuration such as the following would solve this cleanly:

recorder:
  purge_keep_days: 730
  statistics_short_term_keep_days: 90

The exact configuration/API is only illustrative.

This would allow users to independently decide how long they want to retain:

• regular state history
• 5-minute short-term statistics
• hourly long-term statistics

The existing default behavior would not need to change. This could be an optional setting for installations that need different retention policies.

Current limitations

The current implementation makes it difficult to combine long state-history retention with shorter short-term-statistics retention.

For example, my Recorder configuration intentionally keeps general state history for up to 730 days. Noisy entities are selectively reduced to 14, 31, 90, or 366 days using recorder.purge_entities.

This works correctly for their state history, but their short-term statistics continue to follow the global retention period.

In a real installation this has resulted in:

• SQLite database: ~24 GiB
• statistics_short_term: 85,598,745 rows
• oldest short-term statistics: ~730 days
• 80,699,203 rows older than 31 days
• 71,943,789 rows older than 90 days
• statistics_short_term table: ~3,983 MiB
• its two main indexes: ~2,579 MiB
• total storage for short-term statistics and those indexes: ~6.4 GiB

For comparison, the hourly long-term statistics table is only ~389 MiB in the same database.

Reducing purge_keep_days would reduce this significantly, but would also remove regular state history that I intentionally want to retain.

Using recorder.purge_entities does not solve this because it does not purge the corresponding statistics_short_term data.

This behavior was previously reported in home-assistant/core#96254 ("statistics_short_term not deleted on recorder.purge_entities"). That issue was eventually closed as stale/not planned, without an apparent alternative mechanism for this use case.

As a result, there currently appears to be no supported way to keep, for example, two years of selected state history while retaining only 90 days of 5-minute statistics.

Technical benefits

Separating these retention policies would give Recorder significantly better storage scalability without sacrificing long-term historical data.

Short-term statistics have fundamentally different storage characteristics from raw states and hourly long-term statistics. A large installation can accumulate tens of millions of 5-minute statistic rows even though older data is primarily useful at the hourly long-term-statistics resolution.

Benefits would include:

• substantially smaller Recorder databases for installations with many statistics-enabled sensors
• smaller indexes and reduced SQLite storage overhead
• less data for Recorder maintenance operations to process
• allowing users to retain detailed state history where it is valuable without also retaining years of 5-minute statistics
• avoiding unsupported manual database maintenance as the only way to achieve this retention model

In the example above, a 90-day short-term-statistics retention would make approximately 84% of the current 85.6 million short-term-statistics rows eligible for removal, while hourly long-term statistics could remain available indefinitely.

This also preserves flexibility: users who want the existing coupled retention behavior could simply leave the new option unset.

Additional context

Related previous issue:

home-assistant/core#96254

An alternative implementation could be to extend recorder.purge_entities so that it optionally purges matching short-term statistics as well.

However, a separate configurable retention period for statistics_short_term seems more generally useful and predictable. It addresses the underlying issue directly: raw states, 5-minute statistics, and hourly long-term statistics serve different purposes and can have very different storage requirements.

I am happy to provide additional database statistics or test behavior on my installation if that would help evaluate or implement this.

[Petro31]

Vote here https://github.com/orgs/home-assistant/discussions/380