-
Notifications
You must be signed in to change notification settings - Fork 996
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement rollback and scan #59
Conversation
b3759bd
to
8a3ba39
Compare
Signed-off-by: Nick Cameron <nrc@ncameron.org>
Signed-off-by: Nick Cameron <nrc@ncameron.org>
@Connor1996 Please take another look. I've addressed review comments and rebased. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rest LGTM
kv/tikv/storage/commands/rollback.go
Outdated
|
||
// The key has already been committed. This should not happen since the client should never send both | ||
// commit and rollback requests. | ||
return fmt.Errorf("storage: rollback: key has already been committed %v", key) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
add a predefined error?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is an error that should never happen and if it happens in real life should end up in a panic somewhere, rather than an error which is possible and is sent back to the client. Given that, I thought it was better practice in Go to use a string error?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we should return it back to client with setting KeyError.Abort
@@ -54,3 +54,23 @@ func (lock *Lock) IsLockedFor(key []byte, txnStartTs uint64) bool { | |||
} | |||
return lock.TS <= txnStartTs | |||
} | |||
|
|||
// LockedError occurs when a key or keys are locked. The protobuf representation of the locked keys is stored as Info. | |||
type LockedError struct { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
let's move all errors into one file?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This error can't be with the others in commands because that would create an import cycle. I could pull it out into its own file in kvstore, but since there is just one error I thought it was better to put it here. I don't mind pulling it out though if you prefer.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh, seems can't still put it in commands/errors.go
? 🙁let's keep it now, and make a reorganization later. As I assumed, the course would be only left the predefined errors, types and latch in final, so the errors should be in one file, not everywhere.
Signed-off-by: Nick Cameron <nrc@ncameron.org>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
support basic functionality Prewrite/Commit/Rollback/Get/Scan Fill more implementations support coprocessor request. It's runnable now. Most of the code is copied from mocktikv. fix build support concurrent writer support more commands fix bugs support multiple regions and auto split. only resolve lock in region range remove tableScan and indexScan limit Avoid creating iterator too many times. support near seek optimize codec fix check prewrite conflict Use ValidForPrefix avoids conflict. And add message 'try again later' will make the error retryable in TiDB. fix delete range assert fix lock bug and support GC 1. lock key should be raw key rather than mvcc key, or resolve lock may send to another region. 2. support GC. fix scan lock is ignored. Should check pair.Err before check pair.Key. fix rollback command and update options change storage model After this change, lock and latest version is stored together with the same key, old version has a different prefix. During scan, we don't need to traverse every old version. GC is not implement yet. fix and optimize resolve lock fix prewrite keys support GC fix GC bug fix panic rename package name to fork fix build The NewChunk has been removed in TiDB. serialize write operation always use file-io for value log use a wake up channel to save CPU usage. fix reverse scan and limit scan count. rename lock/unlock. Delete instead of SetEntry if UserMeta is maixedDelFlag fix gc panic do not trace already committed error fix panic reduce prewrite conflict possibility and fix resolve lock panic extract acquire lock function and log latency refactor for efficient Resolve request. address comment user defer unlock write requests in batch to avoid exceed max transaction size. And provide an option to set the max transaction size. fix lock bug and implement efficient ScanLock 4k entries in a batch makes prewrite idempotent fix table scan and index scan bugs. makes commit request idempotent fix reverse scan use getter to avoid creating iterator in BatchGet close the DB when received signal. If we set sync write to false, the data may be corrupted if we do not close DB. fix GC bug (#25) * fix GC bug fix the lock unresolvable bug. (#26) * fix lock unresolvable remove key encoding (#27) * remove mv keys, always use raw key fix split bug after removing key encoding (#29) This bug will cause a single region as big as the whole database. Should use region.startKey instead of region.meta.StartKey in split check. return ErrLocked instead of ErrRetryable in Prewrite (#30) introduce lockstore (#31) * introduce lockstore A in-memory lock-free data-structure to store lock. - allows single writer and multiple readers without lock. - memory can be reused after all the entries in a block are deleted. * manual inline getNext add an command line flog to set value log path. (#32) kv: tiny clean up RUnlock and RLock might cost more then a memory allocation, let's keep things simple lock less scope avoid holding txnKeysMu for long time (#35) * avoid holding txnKeysMu for long time move lock out of LSM tree (#36) * move lock out of LSM tree. fix panic introduced by DBReader refactoring (#37) tiny fix Make code more clear Update mvcc.go move acquireLatches method from MVCCStore to regionCtx clear magic number (#41) move write batch out of latch scope (#42) move write batch construction out of latch scope (#43) Tiny refactor (#44) check `badger.ErrKeyNotFound` error returned by `txn.Get`. (#45) fix duplicate prewrite request caused panic. (#47) The lock store only supports insert and delete, if there is a same key, insert would fail and panic. When we meet a duplicate prewrite, we should not write the lock store. avoid `txn.Get` in `Commit` if there is no old version. (#46) unify names (#48) unify iterator style (#49) fix timestamp comparation in KvGet (#50) rename LockStore to MemStore (#51) typo (#52) fix bugs found in high conflicting concurrent transaction. (#53) 1. Fix `Cleanup` mistakenly write the batch to DB instead of lock store and use separate write batch type for Lock and DB to prevent error like this. 2. `Rollback` function should read all keys in lock store before read DB. 3. Fix delete key do actual delete instead of write tombstone. make conflict errors retryable (#54) fix unused import library (#55) fix WaitGroup (#56) After `WaitGroup.Wait` is called, another `Add` call will cause panic, so call `Add` in the Lock to prevent this. * call done after epoch not match split rollbackGC batch (#57) In TPCC benchmark, the number of rollback keys in one minutes are very large, up to hundreds of thousands. It increases latency a lot. Split the rollbackGC batch into small batches to avoid high latency. Make map with capacity (#58) preallocate slices (#59) make source more readable (#60) make it compile with latest tidb (#61) config: add max-procs (#63) support chunk execution (#64) * support chunk execution inline errors.Trace call explicitly (#65) unify Scan function to use closure and clean up code (#66) specify startKey and endKey in iterator options (#67) use Value instead of ValueCopy (#69) clear db reader (#70) optimize count(*) (#71) * optimize count(*) fix compile issue (#72) rename buf to cols (#73) fix build with latest tipb (#74) fix build for removing types.DatumRow (#75) implement closure executor (#76) * implement closure executor handle end key checking implicitly and remove the checking out of hot… (#77) support more executors in closure executor (#78) support the new row format (#79) clean up logs (#80) optimize and refactor encoding and add benchmark (#81) add old value to lock (#82) Avoid one DB.Get in Commit operation. continue handle request if necessary (#84) Rename and cleanup (#85) * use multiple badger DB * pre-split more regions and use 8 DBs * rename project name fix build error and use dep to manage dependencies (#86) * fix build error and use dep to manage dependencies add a flag to control shard-key feature (#88) update dep (#90) add topn closure executor (#89) * add topn closure executor update badger version (#93) use go mod (#94) fix memory leak in lock store (#95) move startTS and commitTS from mvccValue to UserMeta. (#96) fix crash when setting shardKey to false (#97) implement GCCompactionFilter (#98) Disable sharding by default (#99) fix delete panic (#100) upgrade mod (#101)
* Implement rollback Signed-off-by: Nick Cameron <nrc@ncameron.org> * Implement scan request Signed-off-by: Nick Cameron <nrc@ncameron.org>
cc #38
PTAL @Connor1996 @andylokandy @zyh-hust