Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

introduce latest cf #95

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 92 additions & 0 deletions text/0095-add-latest-cf.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Introduce latest cf

- RFC PR: https://github.com/tikv/rfcs/pull/95
- Tracking Issue: https://github.com/tikv/repo/issues/0000

## Summary

Add a cf (column family in RocksDB) named "latest" to store the latest version of keys in MVCC.

## Motivation

As we all know, currently TiKV stores all its data in RocksDB. It creates a cf (column family) named "write" to store all the available versions. The key format looks like following:

```
| key | version |
```

. Version is a 64 bits number and encoded in desc order (larger goes first).

When reading a key with version v0, TiKV is expected to return the largest version v1 that v1 <= v0. Because TiKV has no idea what version is available, so it has to create an iterator and use version v0 to encode a seek key. If any key is hit, then the key should be the requested key.

The procedure is straightforward but expensive. Creating an iterator in RocksDB is not free. Even for point get, an iterator is still necessary. And seek operation is also expensive, almost as expendsive as creating iterator. To avoid seeking too many times, we introduce `near_seek` to TiKV in the early days, which tries `next` several times before fallback to `seek`.

As explained, the reason why seek is necessary is because TiKV has no idea what versions are available. Otherwise it can just use `get` to query the specific key, which is a lot cheaper.

## Detailed design
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about lightning, cdc, br, tiflash's compatibility?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are not part of TiKV. I would like to talk about them in another doc.


TiKV doesn't need to know all existing versions of keys. In fact, most of the time, v0 is larger than any existing versions of keys in TiKV if there is more read than write. So it should be enough to just let TiKV knows the latest version of all keys.

The RFC propose to add a new cf named "latest". When a key is inserted using transaction API, it should update latest cf using the original key without any encoding or version. The value should be similar with the one that is used for updating write cf in the past, but including the corresponding version. If the key exists in the latest cf, then its value should be read and write to write cf using the old format along side with the latest cf update.

For example, supposing there is no key in latest cf. Inserting k1 with version v0 and value foo will insert one key:
- to latest cf, k1 -> (foo, v0 and other meta)

Inserting k1 again with version v1 and value bar will insert two keys:
- to write cf, k1|v0 -> (foo and other meta)
- to latest cf, k1 -> (bar, v1 and other meta)

So all keys in latest cf represent the latest version of all keys.

When all the versions of a key are gced, then it should also delete the key in latest cf only when it matches the last gc key version.

When a key is queried, it should query latest cf using `get` first. If nothing is found or the version is larger than requested, it should query the write cf as fallback. In most case, only one `get` is performed.

When a range scan is triggered, it should scan the latest cf directly. If a larger version is met, it should fallback to seek write cf for that specific key instead. Because only the latest versions are stored in latest cf, so the keys needs to be scanned will be way fewer than the write cf. And in most cases, only one `seek` is performed, and all other operations are `next`.

All of the fallback queries should be performed lazily.

The improvment should be very significant when update keys frequently.

### Compatibility

Because all keys are written to latest cf first, so it will not be compatible with existing write cf as at least one key is missing. To make the them switch easier, Let's introduce a intermediate format that latest value is written to lastest cf and write cf at the same time. Every range that is expecting to upgrade to latest format, it should upgrade to intermediate format first.

```mermaid
graph LR;
origin[Original Format]
inter[Intermediate Format]
latest[Latest Format]
origin --"query using origin way"--> inter --"query using latest way"--> latest
```

Because write cf may not contain the latest change, latest cf should be always queried in all TiKV internal services like GC.

Public APIs should tell TiKV whether latest cf should be used, so that upgrading between format can be seamless to TiKV. Client should ensure only when it updates the range to at least intermediate format will it ask TiKV to query using latest cf.

Take TiDB as an example, it can add a new storage format at table level. Any new table should use the latest format. For existing tables, origin format should be used. However, it can add a new DDL job for table to upgrade the storage format to intermediate format. And only trigger TiKV to use latest cf when the target table is fully upgraded to the intermediate format.

As a new cf is added, it needs to be also included in the raft snapshot between replicas.

### Why use a new cf?

1. The key format is different, using different cf is more efficient.
2. Changing existing cf can bring more compatibility issues than introducing a new one.

## Drawbacks

It introduces a write amplification appearantly. That is also the reason why the access pattern needs to be controlled by client. Client should enable latest cf only when it knows a range of keys are updated very often and can be benificial from the change.

On the other hand, the additional write is just a key in a different cf and a value that is probably not larger than 255 bytes, the overhead may not be very signifiant. More experiments are needed.

## Alternatives

unistore separates the latest version and other versions by adjust file format. So when flushing or compacting, it will make latest versions key be the first part, and the rest in the second part. This approach doesn't have write overhead, but is not backward compatible in TiKV.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about create a history cf to store as many history version as it can, the write cf store the latest several versions, and background thread like GC thread move existed history versions to history cf.
For point get, using RocksDB's user timestamp feature to get the latest version can see, this also can prevent creating iterator.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no compatibility issue for this solution.

Copy link
Member Author

@BusyJay BusyJay Jun 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compaction filter is not reliable, we have seen issue caused by not having compaction in time like tikv/tikv#12729. This proposal doesn't depend on the user timestamp, which is not used in production yet. And can be landed table by table instead of the whole cluster, have less impact on existing code. And this proposal should have better scan performance than relying on compaction filter.

Copy link
Member Author

@BusyJay BusyJay Jun 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added as an alternative with more argument.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using history cf doesn't need the compaction filter, gc just check the history cf sst file's max ts to determine wether to drop the whole sst file.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compaction filter is not used for dropping history sst, but for moving data from write cf to history cf.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compaction filter is not necessary, I think version statistics-based GC is much better.


Another proposal has also been discussed in the past that instead of adding latest cf, adding a history cf to store old versions. All keys are written to write cf first, and then using compaction filter to move all versions except the latest to history cf. This approach delay the additional writes to background job, so may have less impact on the foreground writes. But it has following shortcomings:
- compaction filter is not reliable. The timing it's triggered can be tricky. We have observed issue that introduced by compaction not in time. tikv/tikv#12729.
- compaction filter only works on SST files, versions in memory are still mixed.
- point get still requires seek unless we switch to user timestamp completely, which is not used in production yet.
- If we remove KV WAL completely, writing in compaction can be expensive as it needs to be either ingested as new SST or triggers flush, otherwise restarting TiKV may lose data.

## Unresolved questions