This repository has been archived by the owner on Jun 7, 2021. It is now read-only.
[TRAFODION-2645] First draft of a rewrite of the MDAM costing code #1246
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This set of changes is a first draft of a rewrite of the MDAM costing code.
The rewritten code uses a model much more closely aligned to how the MDAM run-time works. It estimates the number of MDAM probes and fetches directly. I/O cost is estimated differently. I/O cost is not additive across disjuncts, because the more parts of a file that are touched, the more like sequential I/O matters become. On the other hand, the cost of an HBase scan (that is, a begin-key/end-key subset in executor terms) is significant, and its contribution to cost is additive. A knob, MDAM_SUBSET_FACTOR, has been added to tune that cost.
The cost formulas used to determine optimal disjunct prefix are as close as possible to the cost formula used to cost the MDAM scan as a whole. The only thing left out in the former is the I/O cost, as that is not additive. In contrast, in the old code, the costing formulas used for optimal disjunct prefix are quite different than that used for the scan as a whole, and it is hard to see their relationship.
I have done a performance test of the test bed in JIRA TRAFODION-1641, using old and new costing code, and forcing both serial and parallel MDAM plans of various depths, and also simple scan plans. The new code aggregate execution time over that test bed is about 6% better than the old. So the code seems to be at least as good as the old. The new code picks the optimal plan more frequently than the old. There are about eight queries (out of 92) where the old code picks a better plan than the new code.
There is still some testing work to be done on this code. Costing of the inner table of a nested join has not been fully explored yet.
In this check-in, the new costing code is turned off by default. Use CQD MDAM_COSTING_REWRITE 'ON' to turn on the new costing code.
Also included in this set of changes is a fix to logsort: If a missing statistics warning was present, logsort was not sorting the result rows.
Also included in this set of changes is a test script, testMdam.py, which can be used to test the performance of various MDAM plans and determine whether the old or new costing code is picking the better plan.