Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions spiceaidocs/docs/components/data-accelerators/data-refresh.md
Original file line number Diff line number Diff line change
Expand Up @@ -242,3 +242,25 @@ Accelerated datasets can be set to automatically evict time-series data exceedin
Retention policies apply to `full` and `append` refresh modes (not `changes`).

The policy is set using the [`acceleration.retention_check_enabled`](/reference/spicepod/datasets#accelerationretention_check_enabled), [`acceleration.retention_period`](/reference/spicepod/datasets#accelerationretention_period) and [`acceleration.retention_check_interval`](/reference/spicepod/datasets#accelerationretention_check_interval) parameters, along with the [`time_column`](/reference/spicepod/datasets#time_column) and [`time_format`](/reference/spicepod/datasets#time_format) dataset parameters.


## Refresh Jitter
Accelerated datasets can be configured to add a random jitter to the refresh interval. This can be useful to avoid a thundering herd problem where multiple datasets are refreshed at the same time. The jitter is added or subtracted from the refresh interval and is between 0 and `refresh_jitter_max`.

Example:

```yaml
datasets:
- from: spice.ai/eth.recent_blocks
name: eth_recent_blocks
acceleration:
refresh_check_interval: 10s
refresh_jitter_enabled: true
refresh_jitter_max: 1s
```

In this example, the refresh interval will be between 9s and 11s.

Refresh jitter can be configured using the following parameters:
- [`refresh_jitter_enabled`](/reference/spicepod/datasets#accelerationrefresh_jitter_enabled)
- [`refresh_jitter_max`](/reference/spicepod/datasets#accelerationrefresh_jitter_max)
8 changes: 8 additions & 0 deletions spiceaidocs/docs/reference/spicepod/datasets.md
Original file line number Diff line number Diff line change
Expand Up @@ -253,6 +253,14 @@ Required when `acceleration.retention_check_enabled` is `true`.

See [Duration](../duration/index.md)

## `acceleration.refresh_jitter_enabled`

Optional. Enable or disable refresh jitter, defaults to `false`. The refresh jitter adds/substracts a randomized time period from the `refresh_check_interval`.

## `acceleration.refresh_jitter_max`

Optional. The maximum amount of jitter to add to the refresh interval. The jitter is a random value between 0 and `refresh_jitter_max`. Defaults to 10% of `refresh_check_interval`.

## `acceleration.indexes`

Optional. Specify which indexes should be applied to the locally accelerated table. Not supported for in-memory Arrow acceleration engine.
Expand Down