Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implements batch executor handler #4242

Merged
merged 72 commits into from Mar 1, 2019
Merged
Changes from 1 commit
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
6c92004
Extract batch column changes
breeswish Feb 14, 2019
efcc5c3
Rename BatchColumn to VectorValue
breeswish Feb 14, 2019
5e53940
Extract clone_vec_with_capacity into util
breeswish Feb 14, 2019
4048a3c
Extract LazyBatchColumn
breeswish Feb 14, 2019
c104c0f
Rename BatchRows to LazyBatchColumnVec
breeswish Feb 14, 2019
5695a5e
- Remove error from LazyBatchColumnVec
breeswish Feb 14, 2019
ae56bca
Merge branch '___batch_extract/batch_column_4' into ___batch_extract/…
breeswish Feb 14, 2019
a167c66
2018 -> 2019
breeswish Feb 14, 2019
bdfe742
Implement encode functions for VectorValue and LazyBatchColumn(Vec)
breeswish Feb 14, 2019
a52d9d5
Merge remote-tracking branch 'origin/master' into ___batch_extract/ba…
breeswish Feb 15, 2019
ccc7410
Merge branch '___batch_extract/batch_column_1' into ___batch_extract/…
breeswish Feb 15, 2019
272088d
Merge branch '___batch_extract/batch_column_2' into ___batch_extract/…
breeswish Feb 15, 2019
fd5c639
Merge branch '___batch_extract/batch_column_3' into ___batch_extract/…
breeswish Feb 15, 2019
68a06a9
Merge branch '___batch_extract/batch_column_4' into ___batch_extract/…
breeswish Feb 15, 2019
79e0713
Merge branch '___batch_extract/batch_column_5' into ___batch_extract/…
breeswish Feb 15, 2019
58ac02f
Address comments in former PRs
breeswish Feb 15, 2019
f6af534
Merge remote-tracking branch 'origin/master' into ___batch_extract/ba…
breeswish Feb 17, 2019
d54dca6
Merge branch '___batch_extract/batch_column_3' into ___batch_extract/…
breeswish Feb 17, 2019
10871f5
Merge branch '___batch_extract/batch_column_4' into ___batch_extract/…
breeswish Feb 17, 2019
0d72d90
Merge branch '___batch_extract/batch_column_5' into ___batch_extract/…
breeswish Feb 17, 2019
f235470
Fix typo
breeswish Feb 18, 2019
463092f
Merge remote-tracking branch 'origin/master' into ___batch_extract/ba…
breeswish Feb 18, 2019
ec5cbb8
Merge remote-tracking branch 'origin/master' into ___batch_extract/ba…
breeswish Feb 19, 2019
13dee0d
Merge branch '___batch_extract/batch_column_4' into ___batch_extract/…
breeswish Feb 19, 2019
b0f5340
Merge branch '___batch_extract/batch_column_5' into ___batch_extract/…
breeswish Feb 19, 2019
db52fee
Merge commit '55eff8a5fb3a7b447cf987120edf2b80a9ad8e7f' into ___batch…
breeswish Feb 19, 2019
ba12228
Merge branch '___batch_extract/batch_column_5' into ___batch_extract/…
breeswish Feb 19, 2019
d91b410
Address comments about some typos and grammar
breeswish Feb 20, 2019
e6e3222
Merge branch 'master' into ___batch_extract/batch_column_6
breeswish Feb 20, 2019
fb4671d
Provide ScalarValue
breeswish Feb 20, 2019
5d63d39
Introduce batch executor interfaces
breeswish Feb 20, 2019
13768be
Introduce batch handler
breeswish Feb 20, 2019
494bb35
wrap debug_assert functions with debug_assertions
breeswish Feb 20, 2019
c8da030
Include typo fixes from #4215
breeswish Feb 20, 2019
50bcc8b
Merge branch '___batch_extract/batch_column_5' into ___batch_extract/…
breeswish Feb 20, 2019
401d187
Merge branch '___batch_extract/batch_column_6' into ___batch_extract/…
breeswish Feb 20, 2019
dfd5070
Merge branch '___batch_extract/batch_column_t2/1' into ___batch_extra…
breeswish Feb 20, 2019
0d078c3
Merge branch '___batch_extract/batch_column_t2/2' into ___batch_extra…
breeswish Feb 20, 2019
4e55c52
Merge remote-tracking branch 'origin/master' into ___batch_extract/ba…
breeswish Feb 21, 2019
5d83415
Merge branch '___batch_extract/batch_column_6' into ___batch_extract/…
breeswish Feb 21, 2019
a1d3280
Merge branch '___batch_extract/batch_column_t2/1' into ___batch_extra…
breeswish Feb 21, 2019
5c0c5fa
Merge branch '___batch_extract/batch_column_t2/2' into ___batch_extra…
breeswish Feb 21, 2019
1edd230
Merge branch 'master' into ___batch_extract/batch_column_6
breeswish Feb 21, 2019
b2cb081
Encode unsigned int separately
breeswish Feb 24, 2019
3a95816
Merge branch '___batch_extract/batch_column_6' into ___batch_extract/…
breeswish Feb 24, 2019
f5dffe6
Merge branch '___batch_extract/batch_column_t2/1' into ___batch_extra…
breeswish Feb 24, 2019
9e38905
Merge branch '___batch_extract/batch_column_t2/2' into ___batch_extra…
breeswish Feb 24, 2019
1efadc3
Fix format
breeswish Feb 24, 2019
67a083b
Fix format
breeswish Feb 24, 2019
1fdf609
Fix format
breeswish Feb 24, 2019
49cd859
Fix format
breeswish Feb 24, 2019
18280a4
Reset storage change
breeswish Feb 25, 2019
63d3785
Merge branch '___batch_extract/batch_column_6' into ___batch_extract/…
breeswish Feb 25, 2019
b5ca80a
Merge remote-tracking branch 'origin/master' into ___batch_extract/ba…
breeswish Feb 25, 2019
3a0b937
Merge branch '___batch_extract/batch_column_t2/1' into ___batch_extra…
breeswish Feb 25, 2019
3ce53fb
Merge branch '___batch_extract/batch_column_t2/2' into ___batch_extra…
breeswish Feb 25, 2019
ad35f3b
Merge branch 'master' into ___batch_extract/batch_column_t2/2
AndreMouche Feb 26, 2019
ed7ade0
Merge remote-tracking branch 'origin/master' into ___batch_extract/ba…
breeswish Feb 26, 2019
460624f
Add comments for `pub` fields and structures
breeswish Feb 26, 2019
f027ddd
Merge branch '___batch_extract/batch_column_t2/2' into ___batch_extra…
breeswish Feb 26, 2019
b828c9c
Merge branch 'master' into ___batch_extract/batch_column_t2/3
breeswish Feb 27, 2019
db6a981
Merge branch 'master' into ___batch_extract/batch_column_t2/2
breeswish Feb 27, 2019
f8ecdbf
Merge branch 'master' into ___batch_extract/batch_column_t2/2
breeswish Feb 28, 2019
7425a51
Add TODOs for two major concerns during review, so that we can continue.
breeswish Feb 28, 2019
cd6fe54
Merge branch '___batch_extract/batch_column_t2/2' into ___batch_extra…
breeswish Feb 28, 2019
9b7603b
Fix dyn
breeswish Feb 28, 2019
b919dc5
Merge commit 'e2e9625a396cee537f7f9e83631452361e6f3caa' into ___batch…
breeswish Feb 28, 2019
029e061
Add deadline and more comments
breeswish Mar 1, 2019
e514a29
Adds several TODOs for these possible tuning works
breeswish Mar 1, 2019
e5808e1
Merge branch 'master' into ___batch_extract/batch_column_t2/3
breeswish Mar 1, 2019
acbbd33
Merge branch 'master' into ___batch_extract/batch_column_t2/3
breeswish Mar 1, 2019
1cc50fa
Merge branch 'master' into ___batch_extract/batch_column_t2/3
breeswish Mar 1, 2019
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.

Always

Just for now

Add comments for `pub` fields and structures

Signed-off-by: Breezewish <breezewish@pingcap.com>
  • Loading branch information...
breeswish committed Feb 26, 2019
commit 460624f172d7b97c155cfcdc1e8ba6d522de602f
@@ -86,7 +86,13 @@ impl crate::util::AssertSend for BatchExecutorContext {}

impl crate::util::AssertSync for BatchExecutorContext {}

/// The actual inner structure for `BatchExecutorContext`. Normally this structure should be
/// accessed via `BatchExecutorContext` and should not be constructed directly.
pub struct BatchExecutorContextInner {
/// The column info of base physical table schemas.
// TODO: Actually we may need different columns_info for different executors. For example,
// the executor above the aggregation executor's (e.g. Projection Executor) schema should be
// the aggregation executor's schema instead of base schema.
pub columns_info: Vec<ColumnInfo>,

// TODO: This is really a execution config, alhtough called eval config.
@@ -99,20 +105,35 @@ pub struct BatchExecutorContextInner {
/// However they are flowed at once, just before response, instead of each step during execution.
/// Hence they are not covered by this structure. See `BatchExecuteMetaData`.
///
/// TODO: Warnings should be flowed in each function call.
///
/// It is only `Send` but not `Sync` because executor returns its own data copy. However `Send`
/// enables executors to live in different threads.
///
/// It is designed to be used in new generation executors, i.e. executors support batch execution.
/// The old executors will not be refined to return this kind of result.
pub struct BatchExecuteResult {
/// The columns data generated during this invocation. Note that empty column data doesn't mean
/// that there is no more data. See `is_drained`.
pub data: LazyBatchColumnVec,

/// The warnings generated during this invocation.
// TODO: It can be more general, e.g. `ExecuteWarnings` instead of `EvalWarnings`.
// TODO: Should be recorded by row.
pub warnings: EvalWarnings,

/// Whether or not there is no more data.
///
/// This structure is a `Result`. When it is:
/// - `Ok(false)`: The normal case, means that there could be more data. The caller should
/// continue calling `next_batch()` although for each call the returned data may
/// be empty.
/// - `Ok(true)`: Means that the executor is drained and no more data will be returned in
/// future. However there could be some (last) data in the `data` field this
/// time. The caller should NOT call `next_batch()` any more.
/// - `Err(_)`: Means that there is an error when trying to retrieve more data. In this case,
/// the error is returned and the executor is also drained. Similar to
/// `Ok(true)`, there could be some remaining data in the `data` field which is
/// valid data and should be processed. The caller should NOT call `next_batch()`
/// any more.
pub is_drained: Result<bool, Error>,
}

@@ -126,7 +147,6 @@ pub struct BatchExecuteStatistics {

/// Scanning statistics for each CF during execution.
pub cf_stats: crate::storage::Statistics,
// TODO: Find somewhere to report exec count.
}

impl BatchExecuteStatistics {
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.