DRILL-5504: Vector validator to diagnose offset vector issues #832
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| * each batch passed to each iterator. | ||
| */ | ||
| String ENABLE_VECTOR_VALIDATION = "debug.validate_vectors"; | ||
| BooleanValidator ENABLE_VECTOR_VALIDATOR = new BooleanValidator(ENABLE_VECTOR_VALIDATION, true); |
Contributor
There was a problem hiding this comment.
false, by default, here and below.
Contributor
Author
There was a problem hiding this comment.
Good catch. Fixed.
But, that error actually accidentally caught a bug...
| if (! enableValidation) { | ||
| enableValidation = context.getOptionSet().getOption(ExecConstants.ENABLE_ITERATOR_VALIDATOR); | ||
| } | ||
| if (enableValidation) { |
Contributor
There was a problem hiding this comment.
if (AssertionUtil.isAssertionsEnabled() ||
context.getOptionSet().getOption(ExecConstants.ENABLE_ITERATOR_VALIDATOR) { ... }
| private void validateWrapper(VectorWrapper<? extends ValueVector> w) { | ||
| if (w instanceof SimpleVectorWrapper) { | ||
| validateVector(w.getValueVector()); | ||
| } |
Contributor
There was a problem hiding this comment.
You mentioned above that HyperVectorWrapper is not validated. Can you open a ticket for the functionality to-be-implemented in this validator?
Contributor
Author
There was a problem hiding this comment.
Done. See DRILL-5526.
| } | ||
| } | ||
|
|
||
| public void validate() { |
Contributor
There was a problem hiding this comment.
Just a thought. Is there a way to enable these checks (and fail if invalid) for pre-commit tests as well?
Contributor
Author
There was a problem hiding this comment.
Great idea! Added a config option that forces vector validation. Add the following to the pom.xml file in the Surefire options:
{code}
-Ddrill.exec.debug.validate_vectors=true
{code}
Will try this out and enable the checks as a different JIRA ticket and PR.
Contributor
|
+1 Please squash the commits. |
Contributor
Author
|
Commits squashed. |
Validates offset vectors in VarChar and repeated vectors. Validates the special case of repeated VarChar vectors (two layers of offsets.) Provides two new session variables to turn on validation. One enables the existing operator (iterator) validation, the other adds vector validation. This allows validation to occur in a “production” Drill (without restarting Drill with assertions, as previously required.) Unit tests validate the validator. Another test validates the integration, but requires manual steps, so is ignored by default. This version is first-cut: all work is done within a single class. Allows back-porting to an earlier version to solve a specific issues. A revision should move some of the work into generated code (or refactor vectors to allow outside access), since offset vectors appear for each subclass; not on a base class that would allow generic operations. * Added boot-time options to allow enabling vector validation in Maven unit tests. * Code cleanup per suggestions. * Additional (manual) tests for boot-time options and default options.
Contributor
Author
|
Fixed typo in log message and rebased onto latest master. |
sudheeshkatkam
pushed a commit
to sudheeshkatkam/drill
that referenced
this pull request
May 26, 2017
Validates offset vectors in VarChar and repeated vectors. Validates the special case of repeated VarChar vectors (two layers of offsets.) Provides two new session variables to turn on validation. One enables the existing operator (iterator) validation, the other adds vector validation. This allows validation to occur in a “production” Drill (without restarting Drill with assertions, as previously required.) Unit tests validate the validator. Another test validates the integration, but requires manual steps, so is ignored by default. This version is first-cut: all work is done within a single class. Allows back-porting to an earlier version to solve a specific issues. A revision should move some of the work into generated code (or refactor vectors to allow outside access), since offset vectors appear for each subclass; not on a base class that would allow generic operations. * Added boot-time options to allow enabling vector validation in Maven unit tests. * Code cleanup per suggestions. * Additional (manual) tests for boot-time options and default options. closes apache#832
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Validates offset vectors in VarChar and repeated vectors. Validates the
special case of repeated VarChar vectors (two layers of offsets.)
Provides two new session variables to turn on validation. One enables
the existing operator (iterator) validation, the other adds vector
validation. This allows validation to occur in a “production” Drill
(without restarting Drill with assertions, as previously required.)
Unit tests validate the validator. Another test validates the
integration, but requires manual steps, so is ignored by default.
This version is first-cut: all work is done within a single class.
Allows back-porting to an earlier version to solve a specific issues. A
revision should move some of the work into generated code (or refactor
vectors to allow outside access), since offset vectors appear for each
subclass; not on a base class that would allow generic operations.