Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Snowflake Normalization doesn't work with table prefix starting with numbers #9259

Closed
marcosmarxm opened this issue Jan 3, 2022 · 0 comments · Fixed by #9301
Closed

Snowflake Normalization doesn't work with table prefix starting with numbers #9259

marcosmarxm opened this issue Jan 3, 2022 · 0 comments · Fixed by #9301

Comments

@marcosmarxm
Copy link
Member

marcosmarxm commented Jan 3, 2022

Environment

  • Airbyte version: 0.35.2-alha
  • OS Version / Instance: GCP
  • Deployment: Docker
  • Source Connector and version: All Sources
  • Destination Connector and version: Snowflake
  • Severity: Very Low / Low / Medium / High / Critical
  • Step where error happened: Sync Job in Normalization Step

Current Behavior

With incremental normalization running Snowflake destination with table prefix starting with numbers break the code.
This is caused because:

if self.is_incremental_mode(self.destination_sync_mode):
if suffix == "scd":
stg_schema = self.get_schema(True)
stg_table = self.tables_registry.get_file_name(schema, self.json_path, self.stream_name, "stg", truncate_name)
if self.destination_type.value == DestinationType.POSTGRES.value:
# Keep only rows with the max emitted_at to keep incremental behavior
config["post_hook"] = (
f"['delete from {stg_schema}.{stg_table} "
+ f"where {self.airbyte_emitted_at} != (select max({self.airbyte_emitted_at}) "
+ f"from {stg_schema}.{stg_table})']"
)
else:
config["post_hook"] = f"['drop view {stg_schema}.{stg_table}']"

When creating a connection with prefix, eg, 1_stg_table the code will generate a post-hook for Snowflake as:

drop view schema.1_stg_table'

which is invalid.

Expected Behavior

Correct the normalization to work with number table name prefix.

Logs

If applicable, please upload the logs from the failing operation.
For sync jobs, you can download the full logs from the UI by going to the sync attempt page and
clicking the download logs button at the top right of the logs display window.

LOG

replace this with
your long log
output here

Steps to Reproduce

Are you willing to submit a PR?

Remove this with your answer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants