You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
While running the benchmark script for the scenario where the path in config.json is a folder, it throws an error saying no file with a supported extension is found.
Describe the bug
While running the benchmark script for the scenario where the path in
config.json
is a folder, it throws an error saying no file with a supported extension is found.Sample
config.json
:{ "databases": [ { "name": "bigquery", "params": { "metadata" : { "database": "astronomer-dag-authoring", "schema": "tmp_astro" }, "conn_id": "bigquery" } } ], "datasets": [ { "name": "five_gb", "size": "5G", "path": "gs://astro-sdk/benchmark/trimmed/pypi/", "rows": 385817, "conn_id": "bigquery", } ] }
Error log:
ValueError: Missing file extension, cannot automatically determine filetype from path 'gs://astro-sdk/benchmark/trimmed/pypi/'. Please pass the 'filetype' param with the explicit filetype (e.g. csv, ndjson, etc.). [2022-06-16 10:27:49,417] {dagbag.py:334} ERROR - Failed to import: ./dags/evaluate_load_file.py
Expected behaviour
It should have run the loaded dataset for all the files inside the folder.
The text was updated successfully, but these errors were encountered: