Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Schema change detection #257

Closed
alberttwong opened this issue Apr 9, 2024 · 3 comments
Closed

Schema change detection #257

alberttwong opened this issue Apr 9, 2024 · 3 comments
Labels
enhancement New feature or request

Comments

@alberttwong
Copy link

Feature Description

Schema change detection. Of course, it'll fail so the question is many continuous testing?

@alberttwong alberttwong added the enhancement New feature or request label Apr 9, 2024
@bvallier
Copy link

bvallier commented Apr 9, 2024

Is this related to full-refresh mode where an existing schema is present and a new one (e.g. an additional column added) is being materialized? @flarco the default behavior seems to fail in this case, which i didn't expect. I thought the table would be wiped and any new columns would be added to new table, but that didn't seem to be happening. Is this expected?

@flarco
Copy link
Collaborator

flarco commented Apr 9, 2024

hey Brice, if it is full refresh, it should recreate the table with the new column. If incremental, it should add the new column (and not fail). Can you paste your log output with debug mode here?

@bvallier
Copy link

bvallier commented Apr 9, 2024

my apologies - looking closer, this is a dagster issue using SnowflakePandasIOManager...

@slingdata-io slingdata-io locked and limited conversation to collaborators May 15, 2024
@flarco flarco converted this issue into discussion #296 May 15, 2024

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants