The stable row id feature introduces a row id index that fragments as updates occur. The size of this index grows linearly with the number of updates. Just from some rough benchmarking:
| # of Updates |
Size of Index |
| 0 |
125KB |
| 100 |
852KB |
| 1000 |
5MB |
| 10000 |
7.8MB |
| (extrap.) 100K |
75MB |
| (extrap.) 1M |
750MB |
This is primarily a cold-start problem. The row id index generally has to be reloaded each time a new dataset version is loaded into memory and this can get quite slow as the index gets large.
However, the larger indexes are slightly slower to apply as well.
As an escape valve we could create a new task, like compaction, which reclusters the data in row id order. This would heal the row id index without changing the semantic meaning of the row ids (e.g. row 7 would still be row 7) but it would require a complete rewrite of the data (it's basically a table copy at this point).
Another escape valve could be to just reassign row ids. This would break the stable row guarantee (row 7 might now be row 400) but it would very fast.
The stable row id feature introduces a row id index that fragments as updates occur. The size of this index grows linearly with the number of updates. Just from some rough benchmarking:
This is primarily a cold-start problem. The row id index generally has to be reloaded each time a new dataset version is loaded into memory and this can get quite slow as the index gets large.
However, the larger indexes are slightly slower to apply as well.
As an escape valve we could create a new task, like compaction, which reclusters the data in row id order. This would heal the row id index without changing the semantic meaning of the row ids (e.g. row 7 would still be row 7) but it would require a complete rewrite of the data (it's basically a table copy at this point).
Another escape valve could be to just reassign row ids. This would break the stable row guarantee (row 7 might now be row 400) but it would very fast.