DBT project for Airbyte normalization #802
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ChristopheDuong
force-pushed
the
chris/dbt
branch
from
3586fc7
to
cf7b921
Compare
cgardens
reviewed
Nov 4, 2020
airbyte-integrations/bases/base-normalization/normalization/transform_catalog/transform.py
Show resolved
Hide resolved
airbyte-integrations/bases/base-normalization/normalization/transform_catalog/transform.py
Outdated
Show resolved
Hide resolved
airbyte-integrations/bases/base-normalization/normalization/transform_catalog/transform.py
Show resolved
Hide resolved
airbyte-integrations/bases/base-normalization/normalization/transform_catalog/transform.py
Outdated
Show resolved
Hide resolved
…into normalization
jrhizor
suggested changes
Nov 6, 2020
.gitignore
Outdated
| @@ -15,3 +15,8 @@ __pycache__ | |||
| .venv | |||
| .mypy_cache | |||
|
|
|||
| # dbt | |||
| profiles.yml | |||
There was a problem hiding this comment.
do we expect profiles.yml to be in the airbyte git repo at all?
| @@ -31,7 +31,7 @@ | |||
| author_email="contact@airbyte.io", | |||
| url="https://github.com/airbytehq/airbyte", | |||
| packages=setuptools.find_packages(), | |||
| install_requires=["airbyte-protocol", "dbt==0.18.1"], | |||
| install_requires=["airbyte-protocol", "dbt>=0.18.1", "pyyaml"], | |||
There was a problem hiding this comment.
What's the benefit of allowing > versions of dbt? Shouldn't we bump the version to upgrade?
airbyte-integrations/connectors/destination-bigquery/src/main/resources/spec.json
Outdated
Show resolved
Hide resolved
airbyte-integrations/bases/base-normalization/integration_tests/one_recipe.json
Outdated
Show resolved
Hide resolved
airbyte-integrations/bases/base-normalization/dbt-project-template/packages.yml
Outdated
Show resolved
Hide resolved
| @@ -10,6 +10,9 @@ WORKDIR /airbyte/normalization_code | |||
| COPY normalization ./normalization | |||
| COPY setup.py . | |||
| RUN pip install . | |||
| COPY dbt-project-template/macros ./dbt-template/macros | |||
There was a problem hiding this comment.
can you do
COPY dbt-project-template/ ./dbt-template/
Instead?
Make DBT SQL more generic and handle postgres special cases in the macros instead
… destination warehouses
dry run is calling Compiling is translating DBT SQL Models into the warehouse's SQL but doesn't actually run it against the warehouse so it does not perform any changes on the destination. We can use that to check for syntax errors |
michel-tricot
approved these changes
Nov 6, 2020
| {%- endmacro %} | ||
|
|
||
| {% macro snowflake__unnest(array_col) -%} | ||
| -- TODO test this!! not so sure yet... |
| @@ -23,4 +23,5 @@ application { | |||
|
|
|||
| buildImage.dependsOn(assemble) | |||
| buildImage.dependsOn(':airbyte-integrations:bases:base-java:buildImage') | |||
| buildImage.dependsOn(':airbyte-integrations:bases:base-normalization:buildImage') | |||
| @@ -24,4 +24,5 @@ application { | |||
|
|
|||
| buildImage.dependsOn(assemble) | |||
| buildImage.dependsOn(':airbyte-integrations:bases:base-java:buildImage') | |||
| buildImage.dependsOn(':airbyte-integrations:bases:base-normalization:buildImage') | |||
| @@ -24,5 +24,6 @@ application { | |||
| } | |||
|
|
|||
| buildImage.dependsOn(assemble) | |||
| buildImage.dependsOn(':airbyte-integrations:bases:base-normalization:buildImage') | |||
| @@ -43,7 +43,7 @@ | |||
|
|
|||
| private static final Logger LOGGER = LoggerFactory.getLogger(DefaultNormalizationRunner.class); | |||
|
|
|||
| public static final String NORMALIZATION_IMAGE_NAME = "airbyte/normalization:0.1.0"; | |||
| public static final String NORMALIZATION_IMAGE_NAME = "airbyte/normalization:dev"; | |||
There was a problem hiding this comment.
can you create an issue with the proper milestone so we fix that before releasing?
| @@ -49,6 +49,9 @@ def createSpotlessTarget = { pattern -> | |||
| '.mypy_cache', | |||
| '.venv', | |||
| '*.egg-info', | |||
| 'dbt-project-template/macros', | |||
There was a problem hiding this comment.
do you need the three? can't you just do dbt-project-template/ ?
jrhizor
approved these changes
Nov 6, 2020
This was referenced Nov 7, 2020
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
To solve #782, let's make a prototype of conversion from json schemas to DBT models
(running first on BigQuery)