release-2.1: various import/export fixes #30425
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes #29042 Release note: None
Fixes #29280 Release note (bug fix): Correct help text for EXPORT.
Since we add all table descriptor spans to the split list, the end of table 1 is the same as the start of table 2, and thus the start of table 2 would appear in the list twice. Since we now support multiple tables and sequences, we can reduce the size of the split list (and thus the routing table) by removing these duplicate split points. No correctness problems, just a small perf/memory boost. Release note: None
When writing the "done spans" list, use the min key instead of trying
to determine the table start key. Same for at the end. This removes the
possibility of misusing the table span end key as described below. We
may end up changing TableDescriptor.TableSpan, but the change here is
guaranteed to be correct regardless of that.
--
The problem is that when importing a large number of tables (>=56),
regardless of pgdump or mysqldump, we have a table key decoding problem.
When determining split points, we merge two lists: 1) all of the sampled
keys, 2) all of the start and and keys of spans of all table and sequence
descriptors being added. Assume that 100 empty tables (i.e., a PGDUMP
file with 100 `CREATE TABLE` statements) are being imported. We start our
numbering at 55, so tables with ID 55 to 154 will be created. We start
with table ID 55 and get its descriptor span: `/Table/5{5-6}`, or in
bytes: `start: "\xbf", end: "\xc0"`. The end span is computed by making
the start key for a table and calling PrefixEnd on it. PrefixEnd doesn't
know anything about table IDs, it just works on bytes and finds the next
byte, hence `bf` -> `c0`. We can do this for all the tables. Great.
Now move on to the SST writer. When writing SSTs we do a check when done
writing them to see if we are at the end of a table's SSTs. If we are,
we mark the entire table range as done so its progress contribution
is complete (and it isn't imported again if we have to restart). This
works by taking the split point key and, assuming it's a valid table
key prefix, extracts out the table ID. However table ID 109 has an end
span that's not a valid table key prefix (because of the blind call to
PrefixEnd described above). For example, the following code:
```
for i := sqlbase.ID(107); i <= 111; i++ {
desc := sqlbase.TableDescriptor{ID: i}
fmt.Println(desc.TableSpan())
}
```
prints
```
/Table/10{7-8}
/Table/10{8-9}
/Table/109{-/PrefixEnd}
/Table/11{0-1}
/Table/11{1-2}
```
So when the SST writer tries to figure out the table ID on table 109 it
encounters an error because it's not a valid table prefix. This happens
because it wasn't properly encoded with encoding.EncodUvarintAscending,
which does (I guess?) other stuff above that.
Fixes #29348
Release note (bug fix): fix IMPORT of empty or small tables under rare
conditions.
|
This change is |
dt
approved these changes
Sep 19, 2018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport:
Please see individual PRs for details.
/cc @cockroachdb/release