Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug in benchmark evaluate_load_file.py when config.json passes folder path #462

Closed
sunank200 opened this issue Jun 16, 2022 · 0 comments · Fixed by #463
Closed

Bug in benchmark evaluate_load_file.py when config.json passes folder path #462

sunank200 opened this issue Jun 16, 2022 · 0 comments · Fixed by #463
Assignees
Labels
bug Something isn't working priority/high High priority

Comments

@sunank200
Copy link
Contributor

Describe the bug
While running the benchmark script for the scenario where the path in config.json is a folder, it throws an error saying no file with a supported extension is found.

Sample config.json:
{ "databases": [ { "name": "bigquery", "params": { "metadata" : { "database": "astronomer-dag-authoring", "schema": "tmp_astro" }, "conn_id": "bigquery" } } ], "datasets": [ { "name": "five_gb", "size": "5G", "path": "gs://astro-sdk/benchmark/trimmed/pypi/", "rows": 385817, "conn_id": "bigquery", } ] }
Error log:
ValueError: Missing file extension, cannot automatically determine filetype from path 'gs://astro-sdk/benchmark/trimmed/pypi/'. Please pass the 'filetype' param with the explicit filetype (e.g. csv, ndjson, etc.). [2022-06-16 10:27:49,417] {dagbag.py:334} ERROR - Failed to import: ./dags/evaluate_load_file.py

Expected behaviour
It should have run the loaded dataset for all the files inside the folder.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working priority/high High priority
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants