Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

backend: dynamically calculate the maximum auto-inc ID #227

Merged
merged 2 commits into from Aug 30, 2019

Conversation

@kennytm
Copy link
Member

commented Aug 18, 2019

What problem does this PR solve?

Fix #222 (= TOOL-1472).

What is changed and how it works?

Remove the piece of code that pre-calculates the final AUTO_INCREMENT ID. Instead, let the encoder determines the actual value dynamically. This gives a tighter upper bound of the final AUTO_INCREMENT value and avoids overflowing AUTO_INCREMENT columns of shorter integer size (this doesn't work if PRIMARY KEY is not a handle column though, since _tidb_rowid will share the same AUTO_INCREMENT sequence)

Check List

Tests

  • Integration test

Side effects

Related changes

This makes sure the final AUTO_INCREMENT value is exactly the maximum value
of the auto-inc column. Fix #222.
@kennytm kennytm force-pushed the kennytm/dont-overflow-autoinc-if-provided branch from 7bd9d12 to 281ac50 Aug 20, 2019
@kennytm

This comment has been minimized.

Copy link
Member Author

commented Aug 20, 2019

/run-all-tests

@@ -179,6 +180,9 @@ func (kvcodec *tableKVEncoder) Encode(
return nil, logKVConvertFailed(logger, row, j, col.ToInfo(), err)
}
record = append(record, value)
if isAutoIncCol {
kvcodec.tbl.RebaseAutoID(kvcodec.se, value.GetInt64(), false)

This comment has been minimized.

Copy link
@lance6716

lance6716 Aug 21, 2019

Contributor

This RebaseAutoID is called at every row in my understanding, why could this
rebase-per-row gives tighter upper bound than rebase-chunk-max, I think they will eventually reach almost same auto-id(rebase4Unsigned, rebase4Signed, Rebase) and now it's more expensive.

This comment has been minimized.

Copy link
@kennytm

kennytm Aug 21, 2019

Author Member

The change allowing a tighter bound is deleting

for _, engine := range cp.Engines {
for _, chunk := range engine.Chunks {
cp.AllocBase = mathutil.MaxInt64(cp.AllocBase, chunk.Chunk.RowIDMax)
}
}

since RowIDMax is normally much larger than the actual row number.

Yes this could become more expensive, but we should perform a measurement to ensure. I bet the CAS loop is still much cheaper than the actual KV encoding.

This comment has been minimized.

Copy link
@lance6716

lance6716 Aug 22, 2019

Contributor

Now i see rebased value is cast from rowId, which is smaller or equal than row's content. And I agree CAS is lightweight. I don't know if we could move this rebase-per-row outside, such as on chunk's EOF, so there's fewer repeat.

case io.EOF:
break outside

This comment has been minimized.

Copy link
@lance6716

lance6716 Aug 29, 2019

Contributor

CAS is lightweight so maybe we shouldn't bother moving it outside ⬆️

@kennytm

This comment has been minimized.

Copy link
Member Author

commented Aug 29, 2019

@lance6716

This comment has been minimized.

Copy link
Contributor

commented Aug 29, 2019

LGTM

@lance6716 lance6716 added status/LGT1 and removed status/PTAL labels Aug 29, 2019
Copy link
Collaborator

left a comment

LGTM

@GregoryIan GregoryIan added status/LGT2 and removed status/LGT1 labels Aug 30, 2019
@GregoryIan

This comment has been minimized.

Copy link
Collaborator

commented Aug 30, 2019

/run-all-tests

@kennytm kennytm merged commit d49fec6 into master Aug 30, 2019
3 checks passed
3 checks passed
idc-jenkins-ci-lightning/build Jenkins job succeeded.
Details
idc-jenkins-ci-tidb-lightning/test Jenkins job succeeded.
Details
license/cla Contributor License Agreement is signed.
Details
@kennytm kennytm deleted the kennytm/dont-overflow-autoinc-if-provided branch Aug 30, 2019
@GregoryIan GregoryIan referenced this pull request Sep 3, 2019
5 of 5 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.