Warn about catalog manifest discrepancy #91
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
PaddyAlton
changed the title
PaddyAlton
force-pushed
the
5-warn-about-catalog-manifest-discrepancy
branch
from
d0535b5
to
8c5a2c3
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
N.B. I stacked this PR on top of #90 - it contains those changes as well as the relevant ones for this work.
EDIT: #90 was split in two - this is now a stack of three PRs. Confusingly, the order is #90 -> #92 -> #91 (this PR)
An issue related to discrepancies between the DBT manifest and the DBT catalog caused me trouble earlier this year, and was mentioned by some others, too (DBT Slack).
It isn't explicitly included in an open issue, but is fundamentally related to #5 . That issue is about columns in the catalog that are not in the manifest, whereas my problem was about columns in the manifest that are not in the catalog (i.e. incorrectly documented columns).
I have added some warnings for both situations.
If a column in the manifest is not in the catalog, no column data type can be inferred. An uninformative error is raised when its looker type is mapped (caused by attempting to run
None.upper()) - I have instead added an informative Exception, which is raised at the point of data-type inference (instead of returningNonefor the type, which, as far as I can tell, will always lead to a later error).