Skip to content

Add an IndexedTableLoader Extension#9273

Closed
suneet-s wants to merge 1 commit intoapache:masterfrom
suneet-s:index-loader-ext
Closed

Add an IndexedTableLoader Extension#9273
suneet-s wants to merge 1 commit intoapache:masterfrom
suneet-s:index-loader-ext

Conversation

@suneet-s
Copy link
Contributor

@suneet-s suneet-s commented Jan 28, 2020

Create an extension that can load an IndexedTable from a provided
IndexIngestionSpec

The extension attemps to load the IngestionSpec on startup and reads the rows
into an RowBasedIndexedTable

This change makes the LifecycleLock non final for easier testing

Follow up patches:

  • Hook up the RowBasedIndexTable to a DataSource so that it can be queried
  • Refactor the RowBasedIndexTable to accept a row so that it can load the
    index table in just one pass over the rows
  • Introduce an IndexedTableResource so that the tables can be loaded via an
    API instead of just through config
  • Refactor the IngestionSpec so that any spec can provide an iterator over
    the input source. This will allow the extension to read the IndexedTable from
    any format.

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths.
  • added integration tests.
  • been tested in a test Druid cluster.

Create an extension that can load an IndexedTable from a provided
IndexIngestionSpec

The extension attemps to load the IngestionSpec on startup and reads the rows
into an RowBasedIndexedTable

This change makes the LifecycleLock non final for easier testing

Follow up patches:
- Hook up the RowBasedIndexTable to a DataSource so that it can be queried
- Refactor the RowBasedIndexTable to accept a row so that it can load the
index table in just one pass over the rows
- Introduce an IndexedTableResource so that the tables can be loaded via an
API instead of just through config
- Refactor the IngestionSpec so that any spec can provide an iterator over
the input source. This will allow the extension to read the IndexedTable from
any format.
@suneet-s suneet-s closed this Jan 28, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant