Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove create_snowplow_manifest_schema step #30

Closed
bill-warner opened this issue Oct 11, 2021 · 0 comments
Closed

Remove create_snowplow_manifest_schema step #30

bill-warner opened this issue Oct 11, 2021 · 0 comments

Comments

@bill-warner
Copy link
Contributor

Currently

  • The snowplow_web_base_sessions_lifecycle_manifest model's schema is defined by the snowplow__manifest_custom_schema variable. Since this model is a 'proper' dbt model, dbt will create this schema before doing anything else in the run if it doesn't exist already.
  • The snowplow_web_incremental_manifest table's schema is also defined by the snowplow__manifest_custom_schema variable. This however is a manually created table via DDL.
  • Since the manifest schema is created by dbt, when we go to create the snowplow_web_incremental_manifest table, the manifest schema exists.

Potential implementation

  • A user overrides the snowplow_web_base_sessions_lifecycle_manifest model's schema but not by changing the snowplow__manifest_custom_schema variable accordingly (this can be achieved by amending the dbt_project.yml)
  • The snowplow_web_incremental_manifest tables schema is still defined by the snowplow__manifest_custom_schema variable. This schema will no longer be generated automatically by dbt since the snowplow_web_base_sessions_lifecycle_manifest model is not using the same schema. This would cause the CREATE TABLE DDL to fail as the schema doesn't exist.
  • This is why the create_snowplow_manifest_schema step is included as a fail safe.

Issue

  • The adapter.create_schema method I use here to manually create the schema, under the hood calls CREATE SCHEMA IF NOT EXISTS.
  • If the user does not have schema creation permissions this will fail.
  • Despite dbt recommending having such permissions for all dbt users, one of our customers using the package does not allow their dbt users to have such permissions. Rather a super user manually creates all the schemas they need for dbt.
  • This meant every time they ran the package it failed due to this step, despite the required schema already existing.

Fix

  • We should remove the schema creation step to be sure the package will run irrespective of the users permissions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant