Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve error message for unsupported types #7

Closed
rhasson opened this issue May 11, 2023 · 1 comment
Closed

Improve error message for unsupported types #7

rhasson opened this issue May 11, 2023 · 1 comment

Comments

@rhasson
Copy link

rhasson commented May 11, 2023

Created a model for Postgres and received an ugly error.

model code:

{{ config(  materialized='incremental',
            sync = True,
            source = 'POSTGRES',
            options={
            'publication_name': 'upsolver-publication',
            'heartbeat_table': 'upsolver.heartbeat',
            'table_include_list': ('sales.salesreason','sales.salesorderheadersalesreason','sales.salesperson','sales.salesorderheader'),
              'column_exclude_list': ('upsolver.heartbeat.key','upsolver.heartbeat.value'),
              'COMPUTE_CLUSTER': 'SQLake'
            },
            partition_by=[{'field':'$full_table_name', 'type':'string'}]
        )
}}

SELECT * FROM {{ ref('postgres_source_conn') }}

Error:

14:27:34.867256 [error] [Thread-1 (]: Unhandled error while executing model.dbt_demo.royon_cdc_postgres_staging
'postgres'
14:27:34.870240 [debug] [Thread-1 (]: Traceback (most recent call last):
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/dbt/task/base.py", line 367, in safe_run
    result = self.compile_and_execute(manifest, ctx)
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/dbt/task/base.py", line 316, in compile_and_execute
    result = self.run(ctx.node, manifest)
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/dbt/task/base.py", line 415, in run
    return self.execute(compiled_node, manifest)
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/dbt/task/run.py", line 277, in execute
    result = MacroGenerator(
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/dbt/clients/jinja.py", line 329, in __call__
    return self.call_macro(*args, **kwargs)
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/dbt/clients/jinja.py", line 256, in call_macro
    return macro(*args, **kwargs)
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/jinja2/runtime.py", line 763, in __call__
    return self._invoke(arguments, autoescape)
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/jinja2/runtime.py", line 777, in _invoke
    rv = self._func(*arguments)
  File "<template>", line 139, in macro
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/jinja2/sandbox.py", line 393, in call
    return __context.call(__obj, *args, **kwargs)
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/jinja2/runtime.py", line 298, in call
    return __obj(*args, **kwargs)
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/dbt/clients/jinja.py", line 329, in __call__
    return self.call_macro(*args, **kwargs)
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/dbt/clients/jinja.py", line 256, in call_macro
    return macro(*args, **kwargs)
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/jinja2/runtime.py", line 763, in __call__
    return self._invoke(arguments, autoescape)
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/jinja2/runtime.py", line 777, in _invoke
    rv = self._func(*arguments)
  File "<template>", line 42, in macro
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/jinja2/sandbox.py", line 393, in call
    return __context.call(__obj, *args, **kwargs)
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/jinja2/runtime.py", line 298, in call
    return __obj(*args, **kwargs)
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/jinja2/runtime.py", line 763, in __call__
    return self._invoke(arguments, autoescape)
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/jinja2/runtime.py", line 777, in _invoke
    rv = self._func(*arguments)
  File "<template>", line 134, in macro
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/jinja2/sandbox.py", line 393, in call
    return __context.call(__obj, *args, **kwargs)
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/jinja2/runtime.py", line 298, in call
    return __obj(*args, **kwargs)
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/dbt/clients/jinja.py", line 329, in __call__
    return self.call_macro(*args, **kwargs)
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/dbt/clients/jinja.py", line 256, in call_macro
    return macro(*args, **kwargs)
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/jinja2/runtime.py", line 763, in __call__
    return self._invoke(arguments, autoescape)
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/jinja2/runtime.py", line 777, in _invoke
    rv = self._func(*arguments)
  File "<template>", line 34, in macro
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/jinja2/sandbox.py", line 393, in call
    return __context.call(__obj, *args, **kwargs)
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/jinja2/runtime.py", line 298, in call
    return __obj(*args, **kwargs)
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/dbt/adapters/upsolver/impl.py", line 72, in separate_options
    job_options = self.enrich_options(config_options, source, 'job_options')
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/dbt/adapters/upsolver/impl.py", line 78, in enrich_options
    options = self.get_options(source, options_type)
  File "/Users/royhasson/projects/updbt/lib/python3.10/site-packages/dbt/adapters/upsolver/impl.py", line 106, in get_options
    options = Copy_options[source.lower()][options_type]
KeyError: 'postgres'

14:27:34.870968 [debug] [Thread-1 (]: Sending event: {'category': 'dbt', 'action': 'run_model', 'label': '644e8964-8ab0-4498-98c0-fcf80baaf9b2', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108c86920>]}
14:27:34.871521 [error] [Thread-1 (]: 1 of 1 ERROR creating sql incremental model royon.royon_cdc_postgres_staging ... [ERROR in 1.13s]
14:27:34.873177 [debug] [Thread-1 (]: Finished running node model.dbt_demo.royon_cdc_postgres_staging
14:27:34.874611 [debug] [MainThread]: Acquiring new upsolver connection 'master'
14:27:34.875093 [debug] [MainThread]: Connection 'master' was properly closed.
14:27:34.875281 [debug] [MainThread]: Connection 'model.dbt_demo.royon_cdc_postgres_staging' was properly closed.
14:27:34.875933 [info ] [MainThread]: 
14:27:34.876162 [info ] [MainThread]: Finished running 1 incremental model in 0 hours 0 minutes and 17.35 seconds (17.35s).
14:27:34.876435 [debug] [MainThread]: Command end result
14:27:34.883029 [info ] [MainThread]: 
14:27:34.883351 [info ] [MainThread]: Completed with 1 error and 0 warnings:
14:27:34.883529 [info ] [MainThread]: 
14:27:34.883694 [error] [MainThread]: 'postgres'
14:27:34.883849 [info ] [MainThread]: 
14:27:34.884003 [info ] [MainThread]: Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
14:27:34.884236 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108d22800>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108c47c70>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108c47d60>]}
14:27:34.884482 [debug] [MainThread]: Flushing usage events
tanyshak added a commit that referenced this issue May 19, 2023
@tanyshak
improved error message for unsupported types #7
e6abd44
navado added a commit that referenced this issue May 22, 2023
@navado @tanyshak
Support more sources and targets (#14)
9516cde 
* added target options to write to Snowflake target

* addes support for ingestion job to write to Snowflake target #10

* added support for merge/insert to load to other targets #11

* override get strategi to make config options insensitive

* improved error message for unsupported types #7

* added type dict options support

* added add_missing_columns option to snowflake transformation options

* added target_type to the models according to the new syntax

* changed column_transformations data type

* made default Datalake for target_type

* corrected typo in options, made made target_connection required

* v0.2.2

---------

Co-authored-by: Tanya Shemet <tanyshak@gmail.com>
@rhasson
Copy link
Author

rhasson commented Jun 5, 2023

Verified and working

@rhasson rhasson closed this as completed Jun 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@rhasson