[dnm] getting CI run for fk migrations with unconditional downgrades #39503
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This change is |
This commit does the following salient things: - change all in-memory foreign key operations to use the new foreign key representation, which is defined on table descriptor instead of index descriptor. - change all reads of serialized table descriptor protos to unconditionally upgrade themselves from the old descriptor version into the new descriptor version. - change all serialization paths of table descriptors to downgrade themselves from the new descriptor version to the old descriptor version *when the cluster version is less than 19.2*. Note that this commit *is not intended to change* any actual foreign key behavior, such as restrictions on dropping or flexibility in adding. Note further that this commit puts the state of this branch into peril with regards to backup and restore on mixed version clusters, which won't work. This is a known issue and will be resolved before 19.2 is released. See issue cockroachdb#39474. There are a couple of other relevant technical details. New-style foreign keys contain 4 special fields that are designed to help with this upgrade. Two are the "pinned" indexes that the foreign key is related to. These will be the same as the indexes that the foreign keys used to live on in 19.1 and before, if the foreign key was upgraded from an old version, or will be synthesized using the same algorithm as before if being created in 19.2-mixed or later. The other two are "validator" fields that are exact copies of the old 19.1 foreign key representation that a particular foreign key was upgraded from. These will *also* be synthesized by the creation of new foreign keys in 19.2-mixed. These fields are used to validate that we didn't do anything wrong when downgrading a new-style foreign key in the mixed version state. --------------------------------------------------------------------- Here is the "state diagram" of how all of the above works: *19.1* - everybody writes old-style fks, nobody understands new-style fks. *19.2-mixed*: a new 19.2 node enters the cluster, but the cluster version is not yet flipped. - 19.2 nodes always write legacy idx fields when synthesizing table descriptors. - 19.2 nodes always assume the existence of legacy idx fields in all table descriptors. - 19.2 nodes unconditionally upgrade all table descriptors from disk into the new format, preserving both the legacy idx fields and a special "validator" struct that is the entirety of the old-style foreign key. - 19.2 nodes unconditionally downgrade all in-memory table descriptors to the old format, making sure the downgrade was correct by exactly matching the result to the special "validator" struct above. - 19.1 nodes continue on their merry way. No new-style descriptors are ever on disk, so they don't know anything is up. *19.2-final*: all nodes are 19.2, and the upgrade switch is flipped. - 19.2 nodes behave identically to the above, except: - 19.2 nodes do not downgrade in-memory table descriptors to the old format. Release note: None Co-authored-by: Lucy Zhang <lucy@cockroachlabs.com> Co-authored-by: Jordan Lewis <jordan@cockroachlabs.com>
thoszhang
force-pushed
the
fk-migration-with-unconditional-downgrade-that-will-definitely-fail-several-tests-that-we-expect
branch
from
aeb68e8
to
d89bc58
Compare
This commit adds a new logic tests configuration that runs logic tests in the mixed-version cluster configuration that will be visible during a mixed 19.1/19.2 state. Specifically, this means that the "binary version" will be 19.2 but the minimum cluster version will be 19.1 (even though for the purposes of this logic test there will not be any actual 19.1 nodes anywhere). There are a couple of "known issue" logic tests that are disabled / hobbled in the fk and schema_change_in_txn logic test configurations, which necessiated splitting these tests into two nearly identical configs that are run with and without the mixed version configuration. Pending the resolution of cockroachdb#39037, the auto-creation of origin indexes for fks on empty tables doesn't work in mixed-version clusters. Pending the resolution of cockroachdb#37712, validating a foreign key constraint within a transaction containing another schema change on the same table also doesn't work in mixed-version clusters. Finally, there's also a small difference in the order of constraint printing for show create table statements in mixed-version clusters. Release note: None Co-authored-by: Lucy Zhang <lucy@cockroachlabs.com> Co-authored-by: Jordan Lewis <jordan@cockroachlabs.com>
This commit has a solution for all 4 cases that you need to worry about with regards to writing and reading table descriptors in a mixed 19.1/19.2 state. We now: 1. upgrade the descriptors from disk while creating a backup 2. downgrade the descriptors from memory into the backup descriptor if we're in a 19.1/2 mixed state 1. upgrade the descriptors from a backup while preparing to restore 4. downgrade the descriptors from memory onto the disk while restoring a backup if we're in a 19.1/2 mixed state Release note: None
thoszhang
force-pushed
the
fk-migration-with-unconditional-downgrade-that-will-definitely-fail-several-tests-that-we-expect
branch
from
d89bc58
to
150f021
Compare
|
We don't need this anymore. |
jordanlewis
deleted the
fk-migration-with-unconditional-downgrade-that-will-definitely-fail-several-tests-that-we-expect
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
plz do not look at me