Skip to content

Commit

Permalink
Retire xabi due to high disk utilisation rate and corrupt indices
Browse files Browse the repository at this point in the history
`xabi` is the latest STH-backed indexer node that's suffering from:
 * corrupt indices, which results in volatile find response latency, and
 * high disk utilisation growth with no obvious link to ingest rate.

The node has now fully utilised its 60Ti PVC, with the index growth of
14Ti over the last couple of days alone. This behaviour was observed in
other STH-backed instances; see #826.

Remove the node, considering the index records stored by the node are
present by other healthy nodes, its corrupt indices and 100% disk
utilisation at 60Ti (that is close to the 64Ti limit for io2 volumes).
  • Loading branch information
masih committed Nov 14, 2022
1 parent 87bad5c commit bc66306
Show file tree
Hide file tree
Showing 9 changed files with 0 additions and 235 deletions.
Expand Up @@ -16,7 +16,6 @@ spec:
# Use service names local to the namespace over HTTP to avoid
# TLS handshake overhead.
- '--backends=http://vega-indexer:3000/'
- '--backends=http://xabi-indexer:3000/'
- '--backends=http://dido-indexer:3000/'
- '--backends=http://oden-indexer:3000/'
- '--backends=http://kepa-indexer:3000/'
Expand Down
Expand Up @@ -4,7 +4,6 @@ List of individually configurable instances:

| Instance | sth bit-size | IOPS per GiB | Value Codec | Whitelist | `STHBurstRate` | `STHSyncInterval` | Running |
|----------|--------------|--------------|--------------|---------------------|----------------|-------------------|-----------------------------------------------------------------------------------------------------------------------------------------------|
| `xabi` | 30 | 5 | `binary` | all | `8388608` | `1s` | [a51f131e986b9ac3cbfd893e9ebc7669345a25d1](https://github.com/filecoin-project/storetheindex/commit/a51f131e986b9ac3cbfd893e9ebc7669345a25d1) |
| `vega` | 30 | 5 | `binary` | `nft.storage` only | `8388608` | `3s` | [1deb5ed222c428f2f64eeea43f970b05ea844225](https://github.com/filecoin-project/storetheindex/commit/a51f131e986b9ac3cbfd893e9ebc7669345a25d1) |
| `oden` | N/A Pebble | 5 | N/A Pebble | `nft.storage` only | N/A Pebble | N/A Pebble | [9df396fbbc40ca634872a47acae5a6b4008cf2e1](https://github.com/filecoin-project/storetheindex/commit/078d43ca27a0a57f4a568bc67f626ded2a44ecff) |
| `dido` | N/A Pebble | 5 | N/A Pebble | all | N/A Pebble | N/A Pebble | [db83b7c9fab3615621063378fdda568c6e8ba209](https://github.com/filecoin-project/storetheindex/commit/db83b7c9fab3615621063378fdda568c6e8ba209) |
Expand Down
Expand Up @@ -6,7 +6,6 @@ namespace: storetheindex
resources:
- pdb.yaml
- pod-monitor.yaml
- xabi # 30-bit bucket size, binary value codec, 5 IOPS per GiB
- vega # 30-bit bucket size, binary value codec, 5 IOPS per GiB, nft.storage only
- oden # pebble, 5 IOPS per GiB, nft.storage only
- dido # pebble, 5 IOPS per GiB, on us-east2a
Expand Down

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

0 comments on commit bc66306

Please sign in to comment.