fix(ingest/unity): Use assigned metastore if not metastore listed in unity catalog #7446
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
ingestion
asikowitz
reviewed
Feb 27, 2023
| def assigned_metastore(self) -> Optional[Metastore]: | ||
| response: dict = self._unity_catalog_api.get_metastore_summary() | ||
| if response.get("metastore_id") is None: | ||
| logger.info("Not found assinged metastore") |
There was a problem hiding this comment.
Typo on assigned. Also, "Assigned metastore not found" would make a little more sense to me, but I don't feel strongly either way
Comment on lines
180
to
184
| if ( | ||
| assigned_metastore | ||
| and metastore.metastore_id != assigned_metastore.metastore_id | ||
| ): | ||
| metastores.append(metastore) |
There was a problem hiding this comment.
If assigned_metastore is None then we'll get no metastores right? This doesn't seem intended.
You can do not assigned_metastore or metastore.metastore_id != assigned_metastore.metastore_id but it might be simpler to just stick the metastores in a map indexed by their id to avoid duplicates
asikowitz
approved these changes
Feb 27, 2023
Comment on lines
182
to
183
| for id in metastores.keys(): | ||
| metastore = metastores[id] |
There was a problem hiding this comment.
Could be more concisely for id, metastore in metastores.items()
jjoyce0510
approved these changes
Feb 27, 2023
oleg-ruban
pushed a commit
to RChygir/datahub
that referenced
this pull request
Feb 28, 2023
yoonhyejin
pushed a commit
that referenced
this pull request
Mar 3, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Based on oss community it seems like there are cases when list metastore doesn't return anything but we still can get metadata from the metastore which is assigned to the Workspace.
https://datahubspace.slack.com/archives/CUMUWQU66/p1676032592423079
Checklist