Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

load_file operator silently fails if the local file is not existing #467

Closed
utkarsharma2 opened this issue Jun 17, 2022 · 2 comments
Closed
Assignees
Labels
improvement Enhancement or improvement in an existing feature

Comments

@utkarsharma2
Copy link
Collaborator

utkarsharma2 commented Jun 17, 2022

Describe the bug
When we use the load_file operator and pass a local file and the file doesn't exist, the operator does nothing and fails silently.

Version

  • Astro: [e.g. 0.9.2]

To Reproduce
Steps to reproduce the behavior:

  1. Use the load_file operator:
     aql.load_file(  # noqa: F841
            input_file=File(
                path='/tmp/non_existing_file.csv',
            ),
            task_id="load_csv",
            output_table=table_metadata,
            chunk_size=chunk_size,
     )
    
  2. Now the operator will run without any warning or failure making the user think that the operator ran successfully.

Expected behavior
Case 1 (default)

The operator should fail with an error if param if_file_doesnt_exist is set to exception saying that the file doesn't exist.

Case 2

if the if_file_doesnt_exist is set to suppress the operator should not raise an exception.

To accommodate this behaviour added a parameter if_file_doesnt_exist to load_file() operator and get_file() function.

@utkarsharma2 utkarsharma2 self-assigned this Jun 17, 2022
@kaxil kaxil added the improvement Enhancement or improvement in an existing feature label Jun 20, 2022
@kaxil
Copy link
Collaborator

kaxil commented Jun 28, 2022

@utkarsharma2 this can be closed, right?

can you comment which PR closed it plz

@kaxil
Copy link
Collaborator

kaxil commented Jun 28, 2022

closed by #476

@kaxil kaxil closed this as completed Jun 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement Enhancement or improvement in an existing feature
Projects
None yet
Development

No branches or pull requests

2 participants