Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Snowflake stage not deleted when load_file() fails #1262

Closed
magdagultekin opened this issue Nov 17, 2022 · 0 comments · Fixed by #1321
Closed

Snowflake stage not deleted when load_file() fails #1262

magdagultekin opened this issue Nov 17, 2022 · 0 comments · Fixed by #1321
Assignees
Labels
bug Something isn't working priority/medium Medium priority product/python-sdk Label describing products
Milestone

Comments

@magdagultekin
Copy link

Describe the bug
When loading a file to Snowflake with load_file() operator, the stage in Snowflake is not being deleted if the task fails.

Version

  • Astro Runtime: 6.0.3

To Reproduce
Steps to reproduce the behavior:

  1. Write the DAG with the following task, make sure there is some incorrect in the CSV file (e.g., one string instead of expected integer):
s3_to_snowflake = aql.load_file(
    task_id="s3_to_snowflake",
    input_file=File(path=f"s3://{S3_BUCKET_NAME}/{S3_FILE_NAME}", filetype=FileType.CSV),
    output_table=Table(
        conn_id=SNOWFLAKE_CONN_ID,
        metadata=Metadata(database=SNOWFLAKE_DATABASE, schema=SNOWFLAKE_SCHEMA),
        name=SNOWFLAKE_TABLE_NAME,
    ),
    if_exists="replace",
)
  1. Create Snowflake connection called SNOWFLAKE_CONN_ID
  2. Run
  3. See not deleted stage in Snowflake

Expected behavior
A stage should be deleted both when the task succeeds and fails.

@phanikumv phanikumv added this to the 1.3.0 milestone Nov 17, 2022
@phanikumv phanikumv added bug Something isn't working product/python-sdk Label describing products labels Nov 17, 2022
@sunank200 sunank200 added the priority/medium Medium priority label Nov 21, 2022
sunank200 added a commit that referenced this issue Nov 30, 2022
# Description

## What is the current behavior?

Currently the snowflake stage is not being dropped incase of an
exception when calling `load_file`.

closes: #1262

## What is the new behavior?

Drop snowflake stage also when file load failed.

## Does this introduce a breaking change?

No.

### Checklist
- [x] Created tests which fail without the change (if possible)
- [ ] Extended the README / documentation, if necessary

Co-authored-by: rajaths010494 <rajath.srinivasaiah@astronomer.io>
Co-authored-by: Ankit Chaurasia <8670962+sunank200@users.noreply.github.com>
sunank200 pushed a commit that referenced this issue Dec 1, 2022
# Description

## What is the current behavior?

Currently the snowflake stage is not being dropped incase of an
exception when calling `load_file`.

closes: #1262

## What is the new behavior?

Drop snowflake stage also when file load failed.

## Does this introduce a breaking change?

No.

### Checklist
- [x] Created tests which fail without the change (if possible)
- [ ] Extended the README / documentation, if necessary

Co-authored-by: rajaths010494 <rajath.srinivasaiah@astronomer.io>
Co-authored-by: Ankit Chaurasia <8670962+sunank200@users.noreply.github.com>
(cherry picked from commit 7e829af)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working priority/medium Medium priority product/python-sdk Label describing products
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants