feat: Atlas Search Extractor #415
Conversation
mgorsk1
requested review from
allisonsuarez,
dikshathakur3119,
feng-tao,
jinhyukchang and
a team
as code owners
mgorsk1
force-pushed
the
feature/atlas_search_extractor
branch
2 times, most recently
from
9c22f99
to
0604f8f
Compare
|
cc @verdan |
mgorsk1
changed the title
mgorsk1
force-pushed
the
feature/atlas_search_extractor
branch
2 times, most recently
from
e7e3552
to
63e60c4
Compare
| } | ||
|
|
||
| def init(self, conf: ConfigTree) -> None: | ||
| self.conf = conf.with_fallback(AtlasSearchDataExtractor.DEFAULT_CONFIG) |
There was a problem hiding this comment.
AtlasSearchDataExtractor.DEFAULT_CONFIG -> self.DEFAULT_CONFIG ?
|
|
||
| @property | ||
| def entity_type(self) -> str: | ||
| return self.conf.get(AtlasSearchDataExtractor.ENTITY_TYPE_KEY) |
There was a problem hiding this comment.
AtlasSearchDataExtractor.ENTITY_TYPE_KEY -> self.ENTITY_TYPE_KEY
| return list(filter(None, input_list)) | ||
|
|
||
| def _get_driver(self) -> Any: | ||
| return Atlas(host=self.conf.get_string(AtlasSearchDataExtractor.ATLAS_URL_CONFIG_KEY), |
There was a problem hiding this comment.
I wouldn't change it since it uses proper convention for refering class properties. These are not class instance properties and this approach is used consistently throughout all objects (extractors/publishers)
There was a problem hiding this comment.
But you are calling this for this instance only. and not under a static method. 🤷♂️
There was a problem hiding this comment.
Not exactly, please refer to sample_atlas_search_extractor.py to see how those are (or could be) used. Any of those _KEY properties can be used as part of config process for Databuilder Job. It's a general convention used in databuilder afaik and I wouldn't want to make our implementations different.
|
|
||
| LOGGER.info(f'Received count: {count}') | ||
|
|
||
| if count > 0: |
|
@feng-tao ready to merge - is it ok to bump version to 4.0.4 ? |
|
@mgorsk1 sure! Just remember to list all the commits between 4.0.3 and 4.0.4 when you do the release. thanks. |
|
@mgorsk1 could you fix the lint? I would like to push a new version soon. |
Signed-off-by: mgorsk1 <gorskimariusz13@gmail.com>
mgorsk1
force-pushed
the
feature/atlas_search_extractor
branch
from
c5569f9
to
ea4887a
Compare
|
thanks @mgorsk1 ! |
Summary of Changes
This MR introduces extractor to populate Elasticsearch with Atlas data. This way users can use Atlas as metadata proxy while search using ES Proxy.
Tests
todo
Documentation
todo
CheckList
Make sure you have checked all steps below to ensure a timely review.
make test