DRILL-7725: Updates to the EVF2 framework#2073
Closed
paul-rogers wants to merge 2 commits intoapache:masterfrom
Closed
DRILL-7725: Updates to the EVF2 framework#2073paul-rogers wants to merge 2 commits intoapache:masterfrom
paul-rogers wants to merge 2 commits intoapache:masterfrom
Conversation
* Supports internal implicit columns * Better support for standard conversions * Handle several reader corner cases * Simplified file reader
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DRILL-7725: Updates to the EVF2 framework
Description
Includes the refinements needed to move the text (CSV) reader to the upgraded EVF (so-called "EVF2"). To avoid excessive complexity, this PR includes just the EVF2 changes. The text reader integration will appear in a separate PR. At present, no code in master yet uses EVF2.
Supports the internal implicit columns needed for the metadata framework.
Provides better support for standard conversions for text-based readers. A new
FixedReceiveracts to receive text columns when the schema is known up front, as for the CSV reader. (Future versions will include aDynamicReceiverto receive columns by name, as for JSON.)Based on suggestions from Charles, simplified how we handle files for file format plugins. The scan framework takes care of most of the work. It picks up the "compressible" attribute from the plugin, then offers a simple
open()method that takes care of opening the file with the Drill file system, using the compressible option as needed. TheSchemaNegotiatorexposes aFileDescripinterface that handles opening the file, as well as giving access to the block offset and size, etc.Handled a number of corner cases when a reader returns no results, but we do have a schema. Differentiated this from the case where the reader is simply empty: no data and no schema.
Documentation
N/A
Testing
Added new tests and ran all existing tests. In fact, ran all tests with the text reader converted (but the text reader code itself is not included here.)