Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

txn: Add txn state's view #22908

Merged
merged 52 commits into from
May 12, 2021
Merged

Conversation

longfangsong
Copy link
Contributor

@longfangsong longfangsong commented Feb 24, 2021

What problem does this PR solve?

Problem Summary:

Currently, there're no way to view the state of a transaction with SQL, which sometimes makes diagnosing related problems hard.

/cc #24199

What is changed and how it works?

Proposal: TiDB Transaction related views

What's Changed:

Add a (CLUSTER_)TIDB_TRX to view the transaction state.

How it Works:

Just create a memTable, and query the data from all session's TxnState on query.

Use Example

use test; create table t(id int, a varchar(128)); 
insert into t(id, a) values (1, 'abcd'); 
BEGIN; 
select * from t for update; 
select * from information_schema.CLUSTER_TIDB_TRX;

Will show:

+-----------------+--------------------+---------------------+------------------------------------------------------------------+--------+--------------------+------+------+------------+------+------+
| INSTANCE        | ID                 | START_TIME          | DIGEST                                                           | STATE  | WAITING_START_TIME | LEN  | SIZE | SESSION_ID | USER | DB   |
+-----------------+--------------------+---------------------+------------------------------------------------------------------+--------+--------------------+------+------+------------+------+------+
| 127.0.0.1:10080 | 424768545227014155 | 2021-05-07 12:56:48 | a200e35b257452d27bbad7f5cb67435a0f5804848baafbc6674212c917816245 | Normal | NULL               |    1 |   19 |          7 | root | test |
+-----------------+--------------------+---------------------+------------------------------------------------------------------+--------+--------------------+------+------+------------+------+------+
1 row in set (0.00 sec)

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn (Will update after this pr is ready for merge.)

Check List

Tests

  • Unit test

Side effects

  • Performance regression
    • Consumes more CPU
    • Consumes more MEM

Release note

  • Add a (CLUSTER_)TIDB_TRX to view the transaction state.

@ti-chi-bot ti-chi-bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Feb 24, 2021
@github-actions github-actions bot added sig/execution SIG execution sig/sql-infra SIG: SQL Infra labels Feb 24, 2021
@longfangsong longfangsong force-pushed the txn-stat-table branch 3 times, most recently from 71675d8 to db0a2a3 Compare February 24, 2021 08:08
@longfangsong
Copy link
Contributor Author

longfangsong commented Feb 24, 2021

/sig transaction

@ti-chi-bot ti-chi-bot added the sig/transaction SIG:Transaction label Feb 24, 2021
@longfangsong longfangsong force-pushed the txn-stat-table branch 6 times, most recently from 0ce5bad to 198a1a0 Compare February 25, 2021 06:04
@ti-chi-bot ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Feb 25, 2021
@longfangsong longfangsong force-pushed the txn-stat-table branch 3 times, most recently from 0abbe94 to e749058 Compare February 25, 2021 06:47
@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 7, 2021
Signed-off-by: longfangsong <longfangsong@icloud.com>
@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 18, 2021
Signed-off-by: longfangsong <longfangsong@icloud.com>
Signed-off-by: longfangsong <longfangsong@icloud.com>
@longfangsong longfangsong marked this pull request as ready for review March 18, 2021 08:12
@cfzjywxk
Copy link
Contributor

/run-all-tests

session/txn.go Outdated Show resolved Hide resolved
@longfangsong
Copy link
Contributor Author

/run-all-tests

@longfangsong
Copy link
Contributor Author

/cc @MyonKeminta @cfzjywxk
Instead of make memory buffer's field atomic, I collect Len() and Size() and update corresponding fields in LazyTxn atomically on each write of memory buffer, and we have no races detected now.
(The failed intergration tests seems has nothing to do with this pr and will be solved by #24052

count int64
size int64
count int
size int
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the size the number of total bytes? I just realize that if a transaction's size exceeds 4GB, it's size may not fit in a int value. It's out of the topic of this PR though.

Copy link
Contributor

@MyonKeminta MyonKeminta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@MyonKeminta
Copy link
Contributor

Note that there's still an unresolved problem in this PR: the sql digest and the txn state/wait time is not fetched at the same time and without any lock. So it's possible that after getting the txn's state, the transaction starts executing the next statement so another sql digest is fetched. causing inconsistency in the result columns. This still needs to be optimized in the future.

Copy link
Contributor

@cfzjywxk cfzjywxk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@cfzjywxk
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 1015efc

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 12, 2021
@cfzjywxk
Copy link
Contributor

Note that there's still an unresolved problem in this PR: the sql digest and the txn state/wait time is not fetched at the same time and without any lock. So it's possible that after getting the txn's state, the transaction starts executing the next statement so another sql digest is fetched. causing inconsistency in the result columns. This still needs to be optimized in the future.

@MyonKeminta we could record a development task for it.

@MyonKeminta
Copy link
Contributor

Note that there's still an unresolved problem in this PR: the sql digest and the txn state/wait time is not fetched at the same time and without any lock. So it's possible that after getting the txn's state, the transaction starts executing the next statement so another sql digest is fetched. causing inconsistency in the result columns. This still needs to be optimized in the future.

@MyonKeminta we could record a development task for it.

Ok. I've directly recorded it in the tracking issue

@ti-chi-bot
Copy link
Member

/run-all-tests

This bot automatically retries jobs that failed on can merge PRs (send feedback to hi-rustin).

Silence the bot with the /merge cancel comment for consistent failures.

@ti-chi-bot ti-chi-bot merged commit b1d134d into pingcap:master May 12, 2021
@Linda-wang00
Copy link

Linda-wang00 commented May 13, 2021 via email

Howie59 pushed a commit to Howie59/tidb that referenced this pull request May 21, 2021
…ingcap#24052)

* *: fix revoke statement for CURRENT_USER() and refine error message

planner: support set tidb_allow_mpp to `2` or `ENFORCE` to enforce use mpp mode. (pingcap#24516)

store/tikv: remove use of SchemaAmender option in store/tikv (pingcap#24408)

*: the value of tikv-client.store-liveness-timeout should not less than 0 (pingcap#24244)

store/tikv: remove use of EnableAsyncCommit option in store/tikv (pingcap#24462)

txn: Add txn state's view (pingcap#22908)

planner: ignore lock for temporary table of PointGet and BatchPointGet (pingcap#24540)

store/tikv: remove use of ReplicaRead transaction option in store/tikv (pingcap#24409)

store/driver: move error to single package (pingcap#24549)

ddl: add check table compatibility for temporary table (pingcap#24501)

store/tikv: remove use of IsStatenessReadOnly option in store/tikv (pingcap#24464)

store/tikv: change backoff type for missed tiflash peer. (pingcap#24577)

store/tikv: remove use of MatchStoreLabels transaction option in store/tikv (pingcap#24465)

executor, meta: Allocate auto id for global temporary tables (pingcap#24506)

store/tikv: remove use of SampleStep option in store/tikv (pingcap#24461)

executor: add partition pruning tests for adding and dropping partition operations (pingcap#24573)

ddl: forbid partition on temporary mode before put into queue (pingcap#24565)

ddl: speedup test case TestIndexOnMultipleGeneratedColumn (pingcap#24487)

execution: Fix issue 24439 Inconsistent error with MySQL for GRANT CREATE USER ON <specific db>.* (pingcap#24485)

*: fix errcheck (pingcap#24463)

test: make TestExtractStartTs stable (pingcap#24585)

ddl: forbid recover/flashback temporary tables (pingcap#24518)

executor: fix point_get result on clustered index when new-row-format disabled but new-collation enabled (pingcap#24544)

executor: Improve the performance of appending not fixed columns (pingcap#20969)

*: typo fix (pingcap#24564)

planner/core: refresh stale regions in cache for batch cop response (pingcap#24457)

binlog: DML on temporary tables do not write binlog (pingcap#24570)

store/tikv: remove use of GuaranteeLinearizability option in store/tikv (pingcap#24605)

store/tikv: remove use of CollectRuntimeStats option in store/tikv (pingcap#24604)

store/tikv: move Backoffer into a single package (pingcap#24525)

variables: init cte max recursive deeps in a new session (pingcap#24609)

store/tikv: move transaction options out to /kv (pingcap#24619)

store/driver: move backoff driver into single package so we can use i… (pingcap#24624)

server: close the temporary session in HTTP API to avoid memory leak (pingcap#24339)

store/tikv: use latest PD TS plus one as min commit ts (pingcap#24579)

planner: fix incorrect TableDual plan built from nulleq (pingcap#24596)

ranger: fix the case which could have duplicate ranges (pingcap#24590)

 executor, store: Pass the SQL digest down to pessimistic lock request (pingcap#24380)

kv: remove UnionStore interface (pingcap#24625)

*: enable gosimple linter (pingcap#24617)

txn: avoid the gc resolving pessimistic locks of ongoing transactions (pingcap#24601)

util: fix wrong enum building for index range  (pingcap#24632)

sessionctx: change innodb large prefix default (pingcap#24555)

store: fix data race about KVStore.tikvClient (pingcap#24655)

executor, privileges: Add dynamic privileges to SHOW PRIVILEGES (pingcap#24646)

ddl: refactor rule [4/6] (pingcap#24007)

cmd: ddl_test modify retryCnt from 5 to 20 (pingcap#24662)

executor: add correctness tests about direct reading with ORDER BY and LIMIT (pingcap#24455)

store/tikv: remove options from unionstore (pingcap#24629)

planner: fix wrongly check for update statement (pingcap#24614)

store/tikv: remove CompareTS (pingcap#24657)

planner, privilege: Add security enhanced mode part 4 (pingcap#24416)

executor: add some test cases about partition table dynamic-mode with split-region (pingcap#24665)

planner: fix wrong column offsets when processing dynamic pruning for IndexJoin (pingcap#24659)

*: Add security enhanced mode part 3 (pingcap#24412)

store/tikv: resolve ReplicaReadType dependencies (pingcap#24653)

executor: add test cases about partition table with `expression` (pingcap#24628)

tablecodec: fix write wrong prefix index value when collation is ascii_bin/latin1_bin (pingcap#24578)

*: compatibility with staleread (pingcap#24285)

session: test that temporary tables will also be retried (pingcap#24505)

domain, session: Add new sysvarcache to replace global values cache (pingcap#24359)

ddl, transaction: DDL on temporary tables won't affect transactions (pingcap#24534)

*: implement tidb_bounded_staleness built-in function (pingcap#24328)

executor: add correctness tests for partition table with different joins (pingcap#24673)

expression: fix the spelling of word arithmetical (pingcap#24713)

store/copr: balance region for batch cop task (pingcap#24521)

store, metrics: Add metrics for safetTS updating (pingcap#24687)

sem: add tidbredact log to restricted variables (pingcap#24701)

session: fix dml_batch_size doesn't load the global variable (pingcap#24710)

store/tikv: retry TSO RPC (pingcap#24682)

expression, planner: push cast down to control function with enum type. (pingcap#24542)

executor: add correctness tests about IndexMerge (pingcap#24674)

variable: change default for DefDMLBatchSize tidbOptInt64 call (pingcap#24697)

planner: add partitioning pruning tests for range partitioning (pingcap#24554)

*: add option for enum push down (pingcap#24685)

txn: break dependency from store/tikv to tidb/kv cause by TransactionOption (pingcap#24656)

executor: enhancement for ListInDisk(support writing after reading) (pingcap#24379)

kv: move TxnScope into kv (pingcap#24715)

execution: fix the incorrect use of cached plan for point get (pingcap#24749)

executor: add correctness tests about direct reading with indexJoin (pingcap#24497)

variable:  fix sysvar datarace with deep copy (pingcap#24732)

*: Implementing RENAME USER (pingcap#24413)

infoschema, executor: Add the deadlock table (pingcap#24524)

docs: Some proposal for renaming and configurations for Lock View (pingcap#24718)

planner: add range partition boundaries tests with BETWEEN expression (pingcap#24598)

oracle: simplify timestamp utilities (pingcap#24688)

executor: fix wrong enum key in point get (pingcap#24618)

ranger: fix incorrect enum range for xxx_ci collation (pingcap#24661)

executor: add some test cases about dynamic-mode and apply operator (pingcap#24683)

store/tikv: remove Variables.Hook (pingcap#24758)

ddl: speed up the execution time of `TestBackwardCompatibility`. (pingcap#24704)

*: prepare errors for CTE (pingcap#24763)

expression: support cast real/int as real (pingcap#24670)

executor: add table name in log (pingcap#24666)

expression: add builtin function ``json_pretty`` (pingcap#24675)

ddl: make `TestDropLastVisibleColumns` stable (pingcap#24790)

* ddl: make `TestDropLastVisibleColumns` stable

*: support AS OF TIMESTAMP read-only begin statement (pingcap#24740)

executor: Fix unstable TestTiDBLastTxnInfoCommitMode (pingcap#24779)

planner: add tests for partition range boundaries for LT/GT (pingcap#24574)

test: record random seed in TestIssue20658 (pingcap#24782)

store/tikv/retry: define Config instead of BackoffType (pingcap#24692)

config: ignore tiflash when show config (pingcap#24770)

privileges: improve dynamic privs registration and tests (pingcap#24773)

README: remove the link of TiDB Monthly Update (pingcap#24791)

region_cache: filter peers on tombstone or dropped stores (pingcap#24726)

util/stmtsummary: remove import package tikv (pingcap#24776)

ddl: grammar check for create unsupported temporary table (pingcap#24723)

*: update go.etcd.io/bbolt (pingcap#24799)

ddl: speed up the execution time of `ddl test` and `Test Chunk pingcap#7 ddl-other` (pingcap#24780)

executor: remove the unnecessary use of fmt.Sprintf (pingcap#24815)

executor: fix index join panic on prefix index on some cases (pingcap#24568)
@wuhuizuo
Copy link
Contributor

/review default

@ti-chi-bot
Copy link

ti-chi-bot bot commented May 15, 2023

@wuhuizuo:
Review of PR #1234

General

This pull request introduces some changes to the txn package and util package. It adds functionality for tracking transaction information, such as the state of a transaction, the number of entries in memory, and other useful information. It also includes some refactoring of the code in the txn package to improve the locking mechanism and optimize some memory usage.

Specific

txn/lazytxn.go

  • LockKeys - This method adds additional tracking of the state of the transaction and stores a timestamp when a lock is blocked. It also updates the txn info structure's EntriesCount and EntriesSize variables. This method appears to be well-implemented and has helpful changes.
  • reset - The method here changes the receiver to LazyTxn rather than TxnState. This change makes sense given the other changes made in this pull request. The cleanup code looks like mostly renaming of variables and seems to be modified appropriately.
  • KeysNeedToLock - The changes to this method look fine. The receiver has been changed to LazyTxn and the implementation has been updated slightly, but the functionality appears to have remained the same.
  • Info - A new method, this adds helpful functionality to the txn package. It populates and returns a txn info structure, which can be used to display transaction information in an informative way.
  • UpdateEntriesCountAndSize - This method also adds helpful functionality. It updates the number and size of entries in the MemDB.

txn/txninfo/txn_info.go

  • This file adds the txninfo package, which includes some new structs and functions that are used to populate transaction information in a way that is useful for users.

util/processinfo.go

  • SessionManager - the interface has been updated to add a ShowTxnList method that returns an array of txninfo structs.

Conclusion

This pull request adds useful functionality for tracking transaction information, and updates the implementaion to improve the locking mechanism and optimize memory usage. The changes seem well-designed and implemented appropriately. I recommend approving this pull request.

In response to this:

/review default

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sig/execution SIG execution sig/planner SIG: Planner sig/sql-infra SIG: SQL Infra sig/transaction SIG:Transaction size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants