feat: Dremio extractor #377
Conversation
joshthoward
requested review from
allisonsuarez,
dikshathakur3119,
feng-tao and
jinhyukchang
as code owners
|
CI fails |
|
you could put the package like https://github.com/amundsen-io/amundsendatabuilder/blob/master/setup.py#L27 |
There was a problem hiding this comment.
lgtm, could you also update the doc (https://github.com/amundsen-io/amundsendatabuilder#list-of-extractors) with brief information? Would be good to update https://github.com/amundsen-io/amundsen#table-connectors in a later pr as well. thanks
| last_row = row | ||
| columns.append(ColumnMetadata( | ||
| row['col_name'], | ||
| unidecode(row['col_description']) if row['col_description'] else None, |
| last_row['cluster'], | ||
| last_row['schema'], | ||
| last_row['name'], | ||
| unidecode(last_row['description']) if last_row['description'] else None, |
There was a problem hiding this comment.
Good point. I saw this in another extractor, but Neo4j doesn't have issues handling unicode. This is removed.
Signed-off-by: Josh Howard <josh.t.howard@ey.com>
|
@jinhyukchang, @allisonsuarez, @dikshathakur3119 Can I get your feedback when convenient? |
|
cool, thanks, could you have another pr to update the doc for extractor (https://github.com/amundsen-io/amundsendatabuilder#list-of-extractors) ? thanks |
Signed-off-by: Josh Howard <josh.t.howard@ey.com> Co-authored-by: Josh Howard <josh.t.howard@ey.com>
Summary of Changes
The PR adds a Dremio extractor object along with sample usage and tests. The primary file added is:
databuilder/extractor/dremio_metadata_extractor.pyNo new dependency was added. This appears to be in line with the de facto standard of not adding extractor dependencies explicitly to the library's
requirements.txtfile. (e.g. the Big Query extractor raises aModuleNotFoundErrorunless the user explicitly add the dependencies required by the extractor) Please let me know if this is an issue.Tests
A new test was created for the above file:
tests/unit/extractor/test_dremio_metadata_extractor.pyNo other test modifications are present.
Documentation
Doc strings are present, but the following example was also provided:
example/scripts/sample_dremio_data_loader.pyCheckList
Make sure you have checked all steps below to ensure a timely review.
make testPlease let me know if any additional information is needed :)