New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MVCC: memory monitoring #6150
Labels
Milestone
Comments
kyukhin
added
the
srv.info
Collection of maintenance tickets about box.info like interfaces
label
Jun 25, 2021
drewdzzz
added a commit
that referenced
this issue
Nov 24, 2021
Currently, histogram keeps maximum over all observations in order to performance so there is no method to get maximum over contained observations. So this patch introduces method that solves this problem. Part of #6150
drewdzzz
added a commit
that referenced
this issue
Nov 24, 2021
There is no information about owner of txn_stmt when it is being allocated. It makes difficult to track memory consumption for each transaction. That is why we need to pass transaction itself to txn_stmt_new() instead of its region. Part of #6150
drewdzzz
added a commit
that referenced
this issue
Nov 24, 2021
An opportunity to call garbage collector of memtx transaction manager manually allows to understand which garbage cannot be freed. This knowledge can help us to improve garbage collector. Also this opportunity makes it easier to test memory manager of memtx transaction manager. Part of #6150
drewdzzz
added a commit
that referenced
this issue
Nov 24, 2021
Since a transaction has been prepared its garbage (produced stories and replaced tuples) cannot be deleted because they are recognized as used in read-view even if there are no transactions in read-view. This problem makes it difficult to test the memory monitoring system, so this patch solves the problem. Close #6635 Part of #6150
drewdzzz
added a commit
that referenced
this issue
Nov 24, 2021
Since transaction can allocate objects on its own, we need a special memory manager for transactions that will be a base class for all the memory managers for transaction managers. One should not create an instance of tx_memory_manager as an independent unit. It should be used only as a base class. Module tx_memory requires to register transaction with a special method before using tx_memory methods with it. tx_memory contains proxies for all the region and mempool [de]allocation methods instead of region_truncate() - the only way to truncate txn's region is to clear it. Part of #6150
drewdzzz
added a commit
that referenced
this issue
Nov 24, 2021
Memory manager for memtx transaction manager. Derived from tx_memory_manager. Implemented methods to track memtx_story objects and tuples referenced by them, added new statuses of txn allocations special for memtx transaction manager. Part of #6150
drewdzzz
added a commit
that referenced
this issue
Nov 24, 2021
It is difficult to monitor memory consumption for each transcation when some functions allocating on txn's region do not have any information about owner of the region. This patch solve the problem. Part of #6150
drewdzzz
added a commit
that referenced
this issue
Nov 24, 2021
Adopt memtx transaction manager to new memory manager and enable it. All the transactions use memtx_tx_memory_manager because currently is the only transaction manager in tarantool. One can get statistics from memtx_tx_memory_manager via box.stat.memtx.mvcc. Closes #6150
drewdzzz
added a commit
that referenced
this issue
Nov 24, 2021
Adopt memtx transaction manager to new memory manager and enable it. All the transactions use memtx_tx_memory_manager because currently is the only transaction manager in tarantool. One can get statistics from memtx_tx_memory_manager via box.stat.memtx.mvcc. Closes #6150
drewdzzz
added a commit
that referenced
this issue
Nov 24, 2021
Currently, histogram keeps maximum over all observations in order to performance so there is no method to get maximum over contained observations. So this patch introduces method that solves this problem. Part of #6150
drewdzzz
added a commit
that referenced
this issue
Nov 24, 2021
There is no information about owner of txn_stmt when it is being allocated. It makes difficult to track memory consumption for each transaction. That is why we need to pass transaction itself to txn_stmt_new() instead of its region. Part of #6150
drewdzzz
added a commit
that referenced
this issue
Nov 24, 2021
An opportunity to call garbage collector of memtx transaction manager manually allows to understand which garbage cannot be freed. This knowledge can help us to improve garbage collector. Also this opportunity makes it easier to test memory manager of memtx transaction manager. Part of #6150
drewdzzz
added a commit
that referenced
this issue
Nov 24, 2021
Since a transaction has been prepared its garbage (produced stories and replaced tuples) cannot be deleted because they are recognized as used in read-view even if there are no transactions in read-view. This problem makes it difficult to test the memory monitoring system, so this patch solves the problem. Close #6635 Part of #6150
drewdzzz
added a commit
to drewdzzz/tarantool
that referenced
this issue
Apr 3, 2022
Since transaction can allocate objects on its own, we need to track it to collect memory statistics of memtx txm. Txn's use only region allocator so this patch introduces tx_region methods - txns should use it instead of region methods to collect allocation statistics. Part of tarantool#6150 NO_DOC=no visible changes NO_CHANGELOG=no visible changes NO_TEST=already covered
drewdzzz
added a commit
to drewdzzz/tarantool
that referenced
this issue
Apr 3, 2022
This patch introduces memtx_tx_region and memtx_tx_mempool: engineers must use only these proxies to collect statistics. Also this patch introduces box.stat.memtx.mvcc - the way to get memtx mvcc memory statistics. Closes tarantool#6150 @TarantoolBot document Title: Memtx MVCC memory monitoring Introduce memtx MVCC memory monitoring. One can get it with box.stat.memtx.mvcc() method or use index to access a particular statistic. The statistics format: txn: user data: max: 0 avg: 0 total: 0 system data: max: 0 avg: 0 total: 0 statements: max: 0 avg: 0 total: 0 memtx mvcc: tuple stories: potentially in read view: stories count: 0 retained tuples count: 0 stories total: 0 retained tuples total: 0 directly used by txn: stories count: 0 retained tuples count: 0 stories total: 0 retained tuples total: 0 used for tracking: stories count: 0 retained tuples count: 0 stories total: 0 retained tuples total: 0 conflicts: max: 0 avg: 0 total: 0 trackers: max: 0 avg: 0 total: 0
|
Method to get statistics:
New format of statistics, @unera, what do you think? |
ok |
drewdzzz
added a commit
to drewdzzz/tarantool
that referenced
this issue
Apr 7, 2022
Since a transaction has been prepared its garbage (produced stories and replaced tuples) cannot be deleted because they are recognized as used in read-view even if there are no transactions in read-view. This problem makes it difficult to test the memory monitoring system, so this patch solves the problem. Close tarantool#6635 Part of tarantool#6150 NO_DOC=no visible changes NO_CHANGELOG=no visible changes
drewdzzz
added a commit
to drewdzzz/tarantool
that referenced
this issue
Apr 7, 2022
Since transaction can allocate objects on its own, we need to track it to collect memory statistics of memtx txm. Txn's use only region allocator so this patch introduces tx_region methods - txns should use it instead of region methods to collect allocation statistics. Part of tarantool#6150 NO_DOC=no visible changes NO_CHANGELOG=no visible changes NO_TEST=already covered
drewdzzz
added a commit
to drewdzzz/tarantool
that referenced
this issue
Apr 7, 2022
This patch introduces memtx_tx_region and memtx_tx_mempool: engineers must use only these proxies to collect statistics. Also this patch introduces box.stat.memtx.mvcc - the way to get memtx mvcc memory statistics. Closes tarantool#6150 @TarantoolBot document Title: Memtx MVCC memory monitoring Introduce memtx MVCC memory monitoring. One can get it with box.stat.memtx.mvcc() method or use index to access a particular statistic. The statistics format: txn: user data: max: 0 avg: 0 total: 0 system data: max: 0 avg: 0 total: 0 statements: max: 0 avg: 0 total: 0 memtx mvcc: tuple stories: potentially in read view: stories count: 0 retained tuples count: 0 stories total: 0 retained tuples total: 0 directly used by txn: stories count: 0 retained tuples count: 0 stories total: 0 retained tuples total: 0 used for tracking: stories count: 0 retained tuples count: 0 stories total: 0 retained tuples total: 0 conflicts: max: 0 avg: 0 total: 0 trackers: max: 0 avg: 0 total: 0
drewdzzz
added a commit
to drewdzzz/tarantool
that referenced
this issue
Apr 7, 2022
This patch introduces memtx_tx_region and memtx_tx_mempool: engineers must use only these proxies to collect statistics. Also this patch introduces box.stat.memtx.mvcc - the way to get memtx mvcc memory statistics. Closes tarantool#6150 @TarantoolBot document Title: Memtx MVCC memory monitoring Introduce memtx MVCC memory monitoring. One can get it with box.stat.memtx.tx() method or use index to access a particular statistic. The statistics format: txn: statements: max: 0 avg: 0 total: 0 user: max: 0 avg: 0 total: 0 system: max: 0 avg: 0 total: 0 mvcc: trackers: max: 0 avg: 0 total: 0 conflicts: max: 0 avg: 0 total: 0 tuples: tracking: stories: count: 0 total: 0 retained: count: 0 total: 0 used: stories: count: 0 total: 0 retained: count: 0 total: 0 read_view: stories: count: 0 total: 0 retained: count: 0 total: 0
drewdzzz
added a commit
to drewdzzz/tarantool
that referenced
this issue
Apr 7, 2022
There is no information about owner of txn_stmt when it is being allocated. It makes difficult to track memory consumption for each transaction. That is why we need to pass transaction itself to txn_stmt_new() instead of its region. Part of tarantool#6150 NO_DOC=refactoring NO_CHANGELOG=refactoring NO_TEST=refactoring
drewdzzz
added a commit
to drewdzzz/tarantool
that referenced
this issue
Apr 7, 2022
An opportunity to call garbage collector of memtx transaction manager manually allows to understand which garbage cannot be freed. This knowledge can help us to improve garbage collector. Also this opportunity makes it easier to test memtx mvcc memory monitoring. Part of tarantool#6150 NO_DOC=internal feature NO_CHANGELOG=internal feature NO_TEST=internal feature
drewdzzz
added a commit
to drewdzzz/tarantool
that referenced
this issue
Apr 7, 2022
It is difficult to monitor memory consumption for each transcation when some functions allocating on txn's region do not have any information about owner of the region. This patch solve the problem. Part of tarantool#6150 NO_DOC=refactoring NO_CHANGELOG=refactoring NO_TEST=refactoring
drewdzzz
added a commit
to drewdzzz/tarantool
that referenced
this issue
Apr 7, 2022
Since a transaction has been prepared its garbage (produced stories and replaced tuples) cannot be deleted because they are recognized as used in read-view even if there are no transactions in read-view. This problem makes it difficult to test the memory monitoring system, so this patch solves the problem. Close tarantool#6635 Part of tarantool#6150 NO_DOC=no visible changes NO_CHANGELOG=no visible changes
drewdzzz
added a commit
to drewdzzz/tarantool
that referenced
this issue
Apr 7, 2022
Since transaction can allocate objects on its own, we need to track it to collect memory statistics of memtx txm. Txn's use only region allocator so this patch introduces tx_region methods - txns should use it instead of region methods to collect allocation statistics. Part of tarantool#6150 NO_DOC=no visible changes NO_CHANGELOG=no visible changes NO_TEST=already covered
drewdzzz
added a commit
to drewdzzz/tarantool
that referenced
this issue
Apr 7, 2022
This patch introduces memtx_tx_region and memtx_tx_mempool: engineers must use only these proxies to collect statistics. Also this patch introduces box.stat.memtx.mvcc - the way to get memtx mvcc memory statistics. Closes tarantool#6150 @TarantoolBot document Title: Memtx MVCC memory monitoring Introduce memtx MVCC memory monitoring. One can get it with box.stat.memtx.tx() method or use index to access a particular statistic. The statistics format: txn: statements: max: 0 avg: 0 total: 0 user: max: 0 avg: 0 total: 0 system: max: 0 avg: 0 total: 0 mvcc: trackers: max: 0 avg: 0 total: 0 conflicts: max: 0 avg: 0 total: 0 tuples: tracking: stories: count: 0 total: 0 retained: count: 0 total: 0 used: stories: count: 0 total: 0 retained: count: 0 total: 0 read_view: stories: count: 0 total: 0 retained: count: 0 total: 0
drewdzzz
added a commit
to drewdzzz/tarantool
that referenced
this issue
Apr 8, 2022
This patch introduces memtx_tx_region and memtx_tx_mempool: engineers must use only these proxies to collect statistics. Also this patch introduces box.stat.memtx.mvcc - the way to get memtx mvcc memory statistics. Closes tarantool#6150 @TarantoolBot document Title: Memtx MVCC memory monitoring Introduce memtx MVCC memory monitoring. One can get it with box.stat.memtx.tx() method or use index to access a particular statistic. The statistics format: txn: statements: max: 0 avg: 0 total: 0 user: max: 0 avg: 0 total: 0 system: max: 0 avg: 0 total: 0 mvcc: trackers: max: 0 avg: 0 total: 0 conflicts: max: 0 avg: 0 total: 0 tuples: tracking: stories: count: 0 total: 0 retained: count: 0 total: 0 used: stories: count: 0 total: 0 retained: count: 0 total: 0 read_view: stories: count: 0 total: 0 retained: count: 0 total: 0
alyapunov
pushed a commit
that referenced
this issue
Apr 11, 2022
There is no information about owner of txn_stmt when it is being allocated. It makes difficult to track memory consumption for each transaction. That is why we need to pass transaction itself to txn_stmt_new() instead of its region. Part of #6150 NO_DOC=refactoring NO_CHANGELOG=refactoring NO_TEST=refactoring
alyapunov
pushed a commit
that referenced
this issue
Apr 11, 2022
An opportunity to call garbage collector of memtx transaction manager manually allows to understand which garbage cannot be freed. This knowledge can help us to improve garbage collector. Also this opportunity makes it easier to test memtx mvcc memory monitoring. Part of #6150 NO_DOC=internal feature NO_CHANGELOG=internal feature NO_TEST=internal feature
alyapunov
pushed a commit
that referenced
this issue
Apr 11, 2022
It is difficult to monitor memory consumption for each transcation when some functions allocating on txn's region do not have any information about owner of the region. This patch solve the problem. Part of #6150 NO_DOC=refactoring NO_CHANGELOG=refactoring NO_TEST=refactoring
alyapunov
pushed a commit
that referenced
this issue
Apr 11, 2022
Since a transaction has been prepared its garbage (produced stories and replaced tuples) cannot be deleted because they are recognized as used in read-view even if there are no transactions in read-view. This problem makes it difficult to test the memory monitoring system, so this patch solves the problem. Close #6635 Part of #6150 NO_DOC=no visible changes NO_CHANGELOG=no visible changes
alyapunov
pushed a commit
that referenced
this issue
Apr 11, 2022
Since transaction can allocate objects on its own, we need to track it to collect memory statistics of memtx txm. Txn's use only region allocator so this patch introduces tx_region methods - txns should use it instead of region methods to collect allocation statistics. Part of #6150 NO_DOC=no visible changes NO_CHANGELOG=no visible changes NO_TEST=already covered
alyapunov
pushed a commit
that referenced
this issue
Apr 11, 2022
This patch introduces memtx_tx_region and memtx_tx_mempool: engineers must use only these proxies to collect statistics. Also this patch introduces box.stat.memtx.mvcc - the way to get memtx mvcc memory statistics. Closes #6150 @TarantoolBot document Title: Memtx MVCC memory monitoring Introduce memtx MVCC memory monitoring. One can get it with box.stat.memtx.tx() method or use index to access a particular statistic. The statistics format: txn: statements: max: 0 avg: 0 total: 0 user: max: 0 avg: 0 total: 0 system: max: 0 avg: 0 total: 0 mvcc: trackers: max: 0 avg: 0 total: 0 conflicts: max: 0 avg: 0 total: 0 tuples: tracking: stories: count: 0 total: 0 retained: count: 0 total: 0 used: stories: count: 0 total: 0 retained: count: 0 total: 0 read_view: stories: count: 0 total: 0 retained: count: 0 total: 0
CuriousGeorgiy
pushed a commit
to CuriousGeorgiy/tarantool
that referenced
this issue
Apr 12, 2022
There is no information about owner of txn_stmt when it is being allocated. It makes difficult to track memory consumption for each transaction. That is why we need to pass transaction itself to txn_stmt_new() instead of its region. Part of tarantool#6150 NO_DOC=refactoring NO_CHANGELOG=refactoring NO_TEST=refactoring
CuriousGeorgiy
pushed a commit
to CuriousGeorgiy/tarantool
that referenced
this issue
Apr 12, 2022
An opportunity to call garbage collector of memtx transaction manager manually allows to understand which garbage cannot be freed. This knowledge can help us to improve garbage collector. Also this opportunity makes it easier to test memtx mvcc memory monitoring. Part of tarantool#6150 NO_DOC=internal feature NO_CHANGELOG=internal feature NO_TEST=internal feature
CuriousGeorgiy
pushed a commit
to CuriousGeorgiy/tarantool
that referenced
this issue
Apr 12, 2022
It is difficult to monitor memory consumption for each transcation when some functions allocating on txn's region do not have any information about owner of the region. This patch solve the problem. Part of tarantool#6150 NO_DOC=refactoring NO_CHANGELOG=refactoring NO_TEST=refactoring
CuriousGeorgiy
pushed a commit
to CuriousGeorgiy/tarantool
that referenced
this issue
Apr 12, 2022
Since a transaction has been prepared its garbage (produced stories and replaced tuples) cannot be deleted because they are recognized as used in read-view even if there are no transactions in read-view. This problem makes it difficult to test the memory monitoring system, so this patch solves the problem. Close tarantool#6635 Part of tarantool#6150 NO_DOC=no visible changes NO_CHANGELOG=no visible changes
CuriousGeorgiy
pushed a commit
to CuriousGeorgiy/tarantool
that referenced
this issue
Apr 12, 2022
Since transaction can allocate objects on its own, we need to track it to collect memory statistics of memtx txm. Txn's use only region allocator so this patch introduces tx_region methods - txns should use it instead of region methods to collect allocation statistics. Part of tarantool#6150 NO_DOC=no visible changes NO_CHANGELOG=no visible changes NO_TEST=already covered
CuriousGeorgiy
pushed a commit
to CuriousGeorgiy/tarantool
that referenced
this issue
Apr 12, 2022
This patch introduces memtx_tx_region and memtx_tx_mempool: engineers must use only these proxies to collect statistics. Also this patch introduces box.stat.memtx.mvcc - the way to get memtx mvcc memory statistics. Closes tarantool#6150 @TarantoolBot document Title: Memtx MVCC memory monitoring Introduce memtx MVCC memory monitoring. One can get it with box.stat.memtx.tx() method or use index to access a particular statistic. The statistics format: txn: statements: max: 0 avg: 0 total: 0 user: max: 0 avg: 0 total: 0 system: max: 0 avg: 0 total: 0 mvcc: trackers: max: 0 avg: 0 total: 0 conflicts: max: 0 avg: 0 total: 0 tuples: tracking: stories: count: 0 total: 0 retained: count: 0 total: 0 used: stories: count: 0 total: 0 retained: count: 0 total: 0 read_view: stories: count: 0 total: 0 retained: count: 0 total: 0
CuriousGeorgiy
added a commit
to CuriousGeorgiy/tarantool
that referenced
this issue
Sep 23, 2022
test tarantoolgh-6150 test uses Lua assertions which are incomprehensible when failed (the only information provided is 'assertion failed!'), making debugging difficult: replace them with luatest assertions. NO_CHANGELOG=<code health> NO_DOC=<code health>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
So if You reflect on the pt 1 and 2, You'll find that there can be a situation:
So it would be nice to monitor how much memory MVCC uses.
Min/Max/AVG per txn/total
The text was updated successfully, but these errors were encountered: