Schema of partition entries in cache and memtable is updated atomically, causing high latency and possible OOM on large partitions #2577
Assignees
Labels
Milestone
Comments
tgrabiec
added
scale/large partitions
symptom/performance
1 task
|
@tgrabiec ping |
|
Adding keywords for issue search: |
|
#6425 outlines a simpler fix that has narrower scope (still this issue shoud be fixed) |
|
Currently, each partition entry has a chain of incremental versions (mutations). New versions are created when we try to add mutation on top of a snapshot. Snapshots are pointers to versions. When a snapshot is released, we try to merge versions on the spot. If merging is preempted, it will move to background. See We could reuse this mechanism for upgrading schema. We'd maintain an invariant that each partition version conforms to a single schema version. We will store schema_ptr per partition version rather than per partition entry. Version merging would check schemas and change behavior when they don't match. If schemas match we use the old behavior of merging newer to older. When schemas are different, we assume that the newer version has a newer schema and merge older to newer, upgrading the schema on-the-fly. Snapshot readers ( When a read sees that the latest version is not at the data source's current schema, it should force the upgrade by creating an empty version conforming to the data source's current schema and let the same version merging mechanism do the upgrade. This way readers will not keep upgrading on-the-fly indefinitely. |
|
The above solution means that we will pay the cost of moving row entries between row trees during schema upgrade. Maybe that's acceptable. If we wanted to avoid that, we could improve the solution to allow a single partition version to be divided into two regions corresponding to the old and the new schema. We'd have:
Then version merging would first upgrade the old version to the new version, and then merge the two versions like it does today. |
|
Why not simply trigger a dump after a schema change? |
|
A dump of what? |
|
Of the memtable with old schema_ptr. |
Somewhat optimistically setting it to 5.3 (assuming mutation partition will get in to either 5.2 or 5.3). |
michoecho
added a commit
to michoecho/scylla
that referenced
this issue
Mar 7, 2023
Preceding commits in this patch series have extended the MVCC mechanism to allow for versions with different schemas in the same entry/snapshot, with on-the-fly and background schema upgrades to the most recent version in the chain. Given that, we can perform gentle schema upgrades by simply adding an empty version with the target schema to the front of the entry. This patch is intended to be the first and only behaviour-changing patch in the series. Previous patches added code paths for multi-schema snapshots, but never exercised them, because before this patch two different schemas within a single MVCC chain never happened. This patch makes it happen and thus exercises all the code in the series up until now. Fixes scylladb#2577
michoecho
added a commit
to michoecho/scylla
that referenced
this issue
Mar 7, 2023
Preceding commits in this patch series have extended the MVCC mechanism to allow for versions with different schemas in the same entry/snapshot, with on-the-fly and background schema upgrades to the most recent version in the chain. Given that, we can perform gentle schema upgrades by simply adding an empty version with the target schema to the front of the entry. This patch is intended to be the first and only behaviour-changing patch in the series. Previous patches added code paths for multi-schema snapshots, but never exercised them, because before this patch two different schemas within a single MVCC chain never happened. This patch makes it happen and thus exercises all the code in the series up until now. Fixes scylladb#2577
michoecho
added a commit
to michoecho/scylla
that referenced
this issue
Mar 14, 2023
Preceding commits in this patch series have extended the MVCC mechanism to allow for versions with different schemas in the same entry/snapshot, with on-the-fly and background schema upgrades to the most recent version in the chain. Given that, we can perform gentle schema upgrades by simply adding an empty version with the target schema to the front of the entry. This patch is intended to be the first and only behaviour-changing patch in the series. Previous patches added code paths for multi-schema snapshots, but never exercised them, because before this patch two different schemas within a single MVCC chain never happened. This patch makes it happen and thus exercises all the code in the series up until now. Fixes scylladb#2577
michoecho
added a commit
to michoecho/scylla
that referenced
this issue
Mar 14, 2023
Preceding commits in this patch series have extended the MVCC mechanism to allow for versions with different schemas in the same entry/snapshot, with on-the-fly and background schema upgrades to the most recent version in the chain. Given that, we can perform gentle schema upgrades by simply adding an empty version with the target schema to the front of the entry. This patch is intended to be the first and only behaviour-changing patch in the series. Previous patches added code paths for multi-schema snapshots, but never exercised them, because before this patch two different schemas within a single MVCC chain never happened. This patch makes it happen and thus exercises all the code in the series up until now. Fixes scylladb#2577
michoecho
added a commit
to michoecho/scylla
that referenced
this issue
Mar 14, 2023
Preceding commits in this patch series have extended the MVCC mechanism to allow for versions with different schemas in the same entry/snapshot, with on-the-fly and background schema upgrades to the most recent version in the chain. Given that, we can perform gentle schema upgrades by simply adding an empty version with the target schema to the front of the entry. This patch is intended to be the first and only behaviour-changing patch in the series. Previous patches added code paths for multi-schema snapshots, but never exercised them, because before this patch two different schemas within a single MVCC chain never happened. This patch makes it happen and thus exercises all the code in the series up until now. Fixes scylladb#2577
michoecho
added a commit
to michoecho/scylla
that referenced
this issue
May 4, 2023
Preceding commits in this patch series have extended the MVCC mechanism to allow for versions with different schemas in the same entry/snapshot, with on-the-fly and background schema upgrades to the most recent version in the chain. Given that, we can perform gentle schema upgrades by simply adding an empty version with the target schema to the front of the entry. This patch is intended to be the first and only behaviour-changing patch in the series. Previous patches added code paths for multi-schema snapshots, but never exercised them, because before this patch two different schemas within a single MVCC chain never happened. This patch makes it happen and thus exercises all the code in the series up until now. Fixes scylladb#2577
This was referenced May 17, 2023
tgrabiec
added a commit
that referenced
this issue
May 24, 2023
After a schema change, memtable and cache have to be upgraded to the new schema. Currently, they are upgraded (on the first access after a schema change) atomically, i.e. all rows of the entry are upgraded with one non-preemptible call. This is a one of the last vestiges of the times when partition were treated atomically, and it is a well known source of numerous large stalls. This series makes schema upgrades gentle (preemptible). This is done by co-opting the existing MVCC machinery. Before the series, all partition_versions in the partition_entry chain have the same schema, and an entry upgrade replaces the entire chain with a single squashed and upgraded version. After the series, each partition_version has its own schema. A partition entry upgrade happens simply by adding an empty version with the new schema to the head of the chain. Row entries are upgraded to the current schema on-the-fly by the cursor during reads, and by the MVCC version merge ongoing in the background after the upgrade. The series: 1. Does some code cleanup in the mutation_partition area. 2. Adds a schema field to partition_version and removes it from its containers (partition_snapshot, cache_entry, memtable_entry). 3. Adds upgrading variants of constructors and apply() for `row` and its wrappers. 4. Prepares partition_snapshot_row_cursor, mutation_partition_v2::apply_monotonically and partition_snapshot::merge_partition_versions for dealing with heterogeneous version chains. 5. Modifies partition_entry::upgrade to perform upgrades by extending the version chain with a new schema instead of squashing it to a single upgraded version. Fixes #2577 Closes #13761 * github.com:scylladb/scylladb: test: mvcc_test: add a test for gentle schema upgrades partition_version: make partition_entry::upgrade() gentle partition_version: handle multi-schema snapshots in merge_partition_versions mutation_partition_v2: handle schema upgrades in apply_monotonically() partition_version: remove the unused "from" argument in partition_entry::upgrade() row_cache_test: prepare test_eviction_after_schema_change for gentle schema upgrades partition_version: handle multi-schema entries in partition_entry::squashed partition_snapshot_row_cursor: handle multi-schema snapshots partiton_version: prepare partition_snapshot::squashed() for multi-schema snapshots partition_version: prepare partition_snapshot::static_row() for multi-schema snapshots partition_version: add a logalloc::region argument to partition_entry::upgrade() memtable: propagate the region to memtable_entry::upgrade_schema() mutation_partition: add an upgrading variant of lazy_row::apply() mutation_partition: add an upgrading variant of rows_entry::rows_entry mutation_partition: switch an apply() call to apply_monotonically() mutation_partition: add an upgrading variant of rows_entry::apply_monotonically() mutation_fragment: add an upgrading variant of clustering_row::apply() mutation_partition: add an upgrading variant of row::row partition_version: remove _schema from partition_entry::operator<< partition_version: remove the schema argument from partition_entry::read() memtable: remove _schema from memtable_entry row_cache: remove _schema from cache_entry partition_version: remove the _schema field from partition_snapshot partition_version: add a _schema field to partition_version mutation_partition: change schema_ptr to schema& in mutation_partition::difference mutation_partition: change schema_ptr to schema& in mutation_partition constructor mutation_partition_v2: change schema_ptr to schema& in mutation_partition_v2 constructor mutation_partition: add upgrading variants of row::apply() partition_version: update the comment to apply_to_incomplete() mutation_partition_v2: clean up variants of apply() mutation_partition: remove apply_weak() mutation_partition_v2: remove a misleading comment in apply_monotonically() row_cache_test: add schema changes to test_concurrent_reads_and_eviction mutation_partition: fix mixed-schema apply()
|
Party!!!!!!!eleven |
|
I just wanted to point out that the solution presented by #13761 relies on the assumption that reads with on-the-fly upgrades aren't that much more expensive than regular reads. Until the incremental upgrade is done, reads will carry the additional cost of on-the-fly upgrades. If it turns out that this cost is too high, the cluster can become overloaded after the schema change, and the cure will become worse than the disease. AFAIK the implementation of upgrades isn't very efficient. I strongly think we should do some performance testing of schema changes under load to rule out the possibility that after the change upgrades are indeed low-latency, but have an unacceptable throughput cost in exchange. We should do that before we branch the next release. |
|
Reproduced with PackagesScylla version: Kernel Version: Issue description
Describe your issue in detail and steps it took to produce it. ImpactDescribe the impact this issue causes to the user. How frequently does it reproduce?Describe the frequency with how this issue can be reproduced. Installation detailsCluster size: 5 nodes (n2-highmem-16) Scylla Nodes used in this run:
OS / Image: Test: Logs and commands
Logs:
|
|
Reproduced with PackagesScylla version: Kernel Version: Issue description
Describe your issue in detail and steps it took to produce it. ImpactDescribe the impact this issue causes to the user. How frequently does it reproduce?Describe the frequency with how this issue can be reproduced. Installation detailsCluster size: 5 nodes (n2-highmem-16) Scylla Nodes used in this run:
OS / Image: Test: Logs and commands
Logs:
|
|
reproduced in PackagesScylla version: Kernel Version: Issue description
Describe your issue in detail and steps it took to produce it. ImpactDescribe the impact this issue causes to the user. How frequently does it reproduce?Describe the frequency with how this issue can be reproduced. Installation detailsCluster size: 5 nodes (n2-highmem-16) Scylla Nodes used in this run:
OS / Image: Test: Logs and commands
Logs:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
After schema change, entries in memtables and cache are still using the old schema. They're upgraded lazily (but atomically) on access. Schema is currently tracked per-partition. This may badly impact latency if partitions are large.
The text was updated successfully, but these errors were encountered: