tokuftdump utility

A- What is tokuftdump:

It’s an utility that intended to diagnose and look into the fractal tree file (i.e on disk representation of the FT), for more details on how fractal tree works please read this presentation by two of the founders of Tokutek Mike and Martin, check out this blog by my colleague Joe LaFlamme it has a good example that you can use to verify and clarify our claims, and it will not hurt if you can look to this FAQ promise :) This tool could be used in conjunction with above materials to help provide support with useful data , and/or simply understand help you understand how FT works. So the goal of this blog is to explain what and how to use tokuftdump utility to diagnose problems like slow query execution, degraded performance, or simply help you understand how your data is presented in the FT. 1- We have modified --interactive to add mr or MessagesReport, rf or readFile, and modified g or garbage in the interactive mode: When you type : $ tokuftdump --interactive /path/to/fractal-tree-file you get ftdump prompt then type help to see the newly added options/features(on bold): ftdump>help help

header

mr/MessagesReport [NUMBER] Reports messages for the level of the tree you want get more details about

rf/readFile /path/to/ft-file-name Switch to a different FT

node NUMBER

bx OFFSET | block_translation OFFSET

dumpdata 0|1

fragmentation

nodesizes

garbage/g

file OFFSET SIZE [outfilename]

quit

2- We added option --json $ tokuftdump --json /path/to/fractal-tree-file to generate message report of the full fractal tree and store it in a json file for processing by monitoring tools. B- How this newly added features works(with examples)?

--interactive

$tokuftdump --interactive /path/to/fractal-tree-file

1- For g or garbage command it simply was enhanced to report garbage at each leaf node level instead of a number that represents the total garbage on the tree, one of the reasons why we will want to see the garbage at node level instead of the total garbage in the tree is: if the garbage is localised for example in one side or the other of the tree that number may show 5% or less of garbage on the tree but that value at leaf level the number will be like over 69%, so queries like min and max will have to go through a lot of garbage to return the correct value

Examples: 1- Table with no garbage $tokuftdump --interactive /path/to/fractal-tree-file ftdump>g …... Total garbage : 0.011%

2- Table with garbage meaning after running a delete statement that will create holes in the leaf nodes:

mysql or mongo>delete … mysql or mongo>delete … ….. mysql or mongo>update ….

$tokuftdump --interactive /path/to/fractal-tree-file ftdump>g LeafNode# 4 has 31 BasementNodes and 100.0% of the allocated space is garbage LeafNode# 5 has 48 BasementNodes and 15.6% of the allocated space is garbage LeafNode# 6 has 23 BasementNodes and 10.5% of the allocated space is garbage ……. LeafNode# 14 has 1 BasementNodes and 39.4% of the allocated space is garbage LeafNode# 15 has 34 BasementNodes and 54.3% of the allocated space is garbage LeafNode# 16 has 34 BasementNodes and 63.4% of the allocated space is garbage …. LeafNode# 20 has 1 BasementNodes and 69.4% of the allocated space is garbage LeafNode# 23 has 1 BasementNodes and 69.4% of the allocated space is garbage LeafNode# 24 has 42 BasementNodes and 57.7% of the allocated space is garbage LeafNode# 34 has 38 BasementNodes and 30.5% of the allocated space is garbage LeafNode# 35 has 32 BasementNodes and 28.9% of the allocated space is garbage ………….. LeafNode# 838 has 24 BasementNodes and 1.1% of the allocated space is garbage ………… LeafNode# 840 has 24 BasementNodes and 1.0% of the allocated space is garbage ……... LeafNode# 856 has 24 BasementNodes and 0.9% of the allocated space is garbage …………… LeafNode# 3066 has 29 BasementNodes and 2.8% of the allocated space is garbage LeafNode# 3067 has 1 BasementNodes and 2.8% of the allocated space is garbage garbage total size : 3781998453 garbage used size : 3675945897 Total garbage : 2.804% << So this shows clearly that that table doesn’t have much garbage but as we we see in the garbage output it’s localised/skewed

2- mr or MessagesReport command scans all non leaf nodes and report any messages that it finds in the tree.

Examples: 1- Table with no messages, meaning the table is newly created and we insert data, then wait for the cleaner thread(a thread that runs on background and pushed messages down the tree) or you can manually run optimize which will push the messages down.

ftdump>mr to STD output: At height 4 Node Count: 1 Messages: 0 At height 3 Node Count: 6 Messages: 0 At height 2 Node Count: 55 Messages: 0 At height 1 Node Count: 448 Messages: 0 At height 0 Node Count: 5913 Messages: 0

3- Table with messages: ftdump>mr to STD output: At height 4 Node Count: 1 Messages: 0 At height 3 Node Count: 3 Messages: 0 At height 2 Node Count: 30 Messages: 773928 ABORT_ANY(s) 4412 COMMIT_ANY(s) At height 1 Node Count: 244 Messages: 940 INSERT(s) 101253 DELETE_ANY(s) 4785757 ABORT_ANY(s) 294237 COMMIT_ANY(s) At height 0 Node Count: 2577 Messages: 0 << expected as there is no messages on leaf node FT's json file was generated here:FT-1424783463.json

--JSON At shell prompt if you type: $ tokuftdump --json /path/to/fractal-tree-file [/out/file.json] it will generates a json file that could be used by other tools or be sent to tokutek support. This command is similar to running mr level at each level of the tree, Example:

$ tokuftdump --json /path/to/fractal-tree-file FT.json $ more FT.json { "FT":[{ "Level": {"Height":"4", "Nodes":[{"ID":"3","Messages":"0"}]}

} , { "Level": {"Height":"3", "Nodes":[{"ID":"1652","Messages":"0"}, {"ID":"1653","Messages":"0"}, {"ID":"2671","Messages":"0"}]}

} , { "Level": {"Height":"2", "Nodes":[{"ID":"167","Messages":[{},{"INSERT":"0"},{"DELETE":"0"},{},{"ABORT":"0"},{},{"COMMITY":"4412"},{},{"COMMIT_BROADCAST_ALL":"0"},{"COMMIT_BROADCAST_TXN":"0"},{"ABORT_BROADCAST_TXN":"0"},{"INSERT_N OVERWTE":"0"},{"OPTIMIZE":"0"},{"OPTIMIZE_FOR_UPGRADE":"0"},{"UPDATE":"0"},{"UPDATE_BROADCAST_ALL":"0"}]}, {"ID":"168","Messages":"0"}, {"ID":"241","Messages":"0"}, {"ID":"317","Messages":"0"}, {"ID":"406","Messages":"0"}, {"ID":"509","Messages":"0"}, {"ID":"609","Messages":"0"}, {"ID":"728","Messages":"0"}, {"ID":"832","Messages":"0"}, {"ID":"940","Messages":"0"}, {"ID":"1028","Messages":"0"}, {"ID":"1128","Messages":"0"}, {"ID":"1244","Messages":"0"}, {"ID":"1337","Messages":"0"}, {"ID":"1443","Messages":"0"}, {"ID":"1562","Messages":"0"}, {"ID":"1651","Messages":"0"}, {"ID":"1779","Messages":"0"}, {"ID":"1870","Messages":"0"}, {"ID":"1956","Messages":"0"}, {"ID":"2091","Messages":"0"}, {"ID":"2164","Messages":"0"}, {"ID":"2508","Messages":[{},{"INSERT":"0"},{"DELETE":"0"},{},{"ABORT":"102937"},{},{"COMMITY":"0"},{},{"COMMIT_BROADCAST_ALL":"0"},{"COMMIT_BROADCAST_TXN":"0"},{"ABORT_BROADCAST_TXN":"0"},{"INSERT_NOVERWTE ":"0"},{"OPTIMIZE":"0"},{"OPTIMIZE_FOR_UPGRADE":"0"},{"UPDATE":"0"},{"UPDATE_BROADCAST_ALL":"0"}]}, {"ID":"2596","Messages":[{},{"INSERT":"0"},{"DELETE":"0"},{},{"ABORT":"105495"},{},{"COMMITY":"0"},{},{"COMMIT_BROADCAST_ALL":"0"},{"COMMIT_BROADCAST_TXN":"0"},{"ABORT_BROADCAST_TXN":"0"},{"INSERT_NOVERWTE ":"0"},{"OPTIMIZE":"0"},{"OPTIMIZE_FOR_UPGRADE":"0"},{"UPDATE":"0"},{"UPDATE_BROADCAST_ALL":"0"}]}, {"ID":"2669","Messages":[{},{"INSERT":"0"},{"DELETE":"0"},{},{"ABORT":"116428"},{},{"COMMITY":"0"},{},{"COMMIT_BROADCAST_ALL":"0"},{"COMMIT_BROADCAST_TXN":"0"},{"ABORT_BROADCAST_TXN":"0"},{"INSERT_NOVERWTE ":"0"},{"OPTIMIZE":"0"},{"OPTIMIZE_FOR_UPGRADE":"0"},{"UPDATE":"0"},{"UPDATE_BROADCAST_ALL":"0"}]}, {"ID":"2743","Messages":[{},{"INSERT":"0"},{"DELETE":"0"},{},{"ABORT":"75795"},{},{"COMMITY":"0"},{},{"COMMIT_BROADCAST_ALL":"0"},{"COMMIT_BROADCAST_TXN":"0"},{"ABORT_BROADCAST_TXN":"0"},{"INSERT_NOVERWTE" :"0"},{"OPTIMIZE":"0"},{"OPTIMIZE_FOR_UPGRADE":"0"},{"UPDATE":"0"},{"UPDATE_BROADCAST_ALL":"0"}]}, {"ID":"2816","Messages":[{},{"INSERT":"0"},{"DELETE":"0"},{},{"ABORT":"101279"},{},{"COMMITY":"0"},{},{"COMMIT_BROADCAST_ALL":"0"},{"COMMIT_BROADCAST_TXN":"0"},{"ABORT_BROADCAST_TXN":"0"},{"INSERT_NOVERWTE ":"0"},{"OPTIMIZE":"0"},{"OPTIMIZE_FOR_UPGRADE":"0"},{"UPDATE":"0"},{"UPDATE_BROADCAST_ALL":"0"}]}, {"ID":"2889","Messages":[{},{"INSERT":"0"},{"DELETE":"0"},{},{"ABORT":"77641"},{},{"COMMITY":"0"},{},{"COMMIT_BROADCAST_ALL":"0"},{"COMMIT_BROADCAST_TXN":"0"},{"ABORT_BROADCAST_TXN":"0"},{"INSERT_NOVERWTE" :"0"},{"OPTIMIZE":"0"},{"OPTIMIZE_FOR_UPGRADE":"0"},{"UPDATE":"0"},{"UPDATE_BROADCAST_ALL":"0"}]}, {"ID":"2962","Messages":[{},{"INSERT":"0"},{"DELETE":"0"},{},{"ABORT":"79335"},{},{"COMMITY":"0"},{},{"COMMIT_BROADCAST_ALL":"0"},{"COMMIT_BROADCAST_TXN":"0"},{"ABORT_BROADCAST_TXN":"0"},{"INSERT_NOVERWTE" :"0"},{"OPTIMIZE":"0"},{"OPTIMIZE_FOR_UPGRADE":"0"},{"UPDATE":"0"},{"UPDATE_BROADCAST_ALL":"0"}]}, {"ID":"3035","Messages":[{},{"INSERT":"0"},{"DELETE":"0"},{},{"ABORT":"115018"},{},{"COMMITY":"0"},{},{"COMMIT_BROADCAST_ALL":"0"},{"COMMIT_BROADCAST_TXN":"0"},{"ABORT_BROADCAST_TXN":"0"},{"INSERT_NOVERWTE ":"0"},{"OPTIMIZE":"0"},{"OPTIMIZE_FOR_UPGRADE":"0"},{"UPDATE":"0"},{"UPDATE_BROADCAST_ALL":"0"}]}]}

}

…..

}