Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix] Remove raw_code from all_results column in models #351

Merged
merged 1 commit into from
Jun 2, 2023

Conversation

jared-rimmer
Copy link
Member

Overview

  • In a recent release we added an all_results column to models which contains all the elements from the manifest for each model.
  • This caused issues with some DWHs because the records started to exceed limits.
  • As a workaround we have introduced batching of records in some previous PRs.
  • However, I think the raw sql is a big part of what is making the records large in the first place and doesn't need to be stored.
  • In a future commit it might be worth revoking more elements of this field / the field entirely and moving back to parsing out elements and adding new columns.

Update type - breaking / non-breaking

  • Minor bug fix
  • Documentation improvements
  • Quality of Life improvements
  • New features (non-breaking change)
  • New features (breaking change)
  • Other (non-breaking change)
  • Other (breaking change)

What does this solve?

Outstanding questions

What databases have you tested with?

  • Snowflake
  • Google BigQuery
  • Databricks
  • Spark
  • N/A

@jared-rimmer jared-rimmer temporarily deployed to Approve Integration Tests June 1, 2023 13:51 — with GitHub Actions Inactive
@jared-rimmer jared-rimmer temporarily deployed to Approve Integration Tests June 1, 2023 13:51 — with GitHub Actions Inactive
@jared-rimmer jared-rimmer temporarily deployed to Approve Integration Tests June 1, 2023 13:51 — with GitHub Actions Inactive
@jared-rimmer jared-rimmer temporarily deployed to Approve Integration Tests June 1, 2023 13:53 — with GitHub Actions Inactive
@jared-rimmer jared-rimmer temporarily deployed to Approve Integration Tests June 1, 2023 13:53 — with GitHub Actions Inactive
@jared-rimmer jared-rimmer temporarily deployed to Approve Integration Tests June 1, 2023 13:53 — with GitHub Actions Inactive
@jared-rimmer jared-rimmer temporarily deployed to Approve Integration Tests June 1, 2023 13:53 — with GitHub Actions Inactive
@jared-rimmer jared-rimmer temporarily deployed to Approve Integration Tests June 1, 2023 13:53 — with GitHub Actions Inactive
@jared-rimmer jared-rimmer temporarily deployed to Approve Integration Tests June 1, 2023 13:53 — with GitHub Actions Inactive
@jared-rimmer jared-rimmer requested a review from glsdown June 1, 2023 13:56
@glsdown glsdown merged commit 24aa4da into main Jun 2, 2023
@glsdown glsdown deleted the fix/remove-raw-code-from-upload-models-all-results branch June 2, 2023 12:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants