Skip to content

Commit

Permalink
Check for filetype only for pattern and fix coverage (#872)
Browse files Browse the repository at this point in the history
# Description
## What is the current behavior?
<!-- Please describe the current behavior that you are modifying. -->
Currently is filetype is passed then pattern doesn't need to be resolved
to check native autodetect

<!--
Issues are required for both bug fixes and features.
Reference it using one of the following:

closes: #ISSUE
related: #ISSUE
-->
closes: #873


## What is the new behavior?
<!-- Please describe the behavior or changes that are being added by
this PR. -->

- Create file type in case not provided before checking for native
autodetect schema


## Does this introduce a breaking change?
No

### Checklist
- [x] Created tests which fail without the change (if possible)
- [x] Extended the README / documentation, if necessary

Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
  • Loading branch information
sunank200 and kaxil committed Sep 19, 2022
1 parent 0a39c84 commit a2dc914
Show file tree
Hide file tree
Showing 2 changed files with 27 additions and 3 deletions.
10 changes: 7 additions & 3 deletions python-sdk/src/astro/databases/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
from astro.exceptions import DatabaseCustomError, NonExistentTableException
from astro.files import File, resolve_file_path_pattern
from astro.files.types import create_file_type
from astro.files.types.base import FileType as FileTypeConstants
from astro.settings import LOAD_TABLE_AUTODETECT_ROWS_COUNT, SCHEMA
from astro.sql.table import BaseTable, Metadata
from pandas.io.sql import SQLDatabase
Expand Down Expand Up @@ -744,11 +745,14 @@ def check_schema_autodetection_is_supported(self, source_file: File) -> bool:
source_file.location.location_type
)

source_filetype = create_file_type(
path=source_file.path, filetype=source_file.type.name
source_filetype = (
source_file
if isinstance(source_file.type, FileTypeConstants)
else create_file_type(path=source_file.path, filetype=source_file.type) # type: ignore
)

is_source_filetype_supported = (
(source_filetype in filetype_supported.get("filetype")) # type: ignore
(source_filetype.type.name in filetype_supported.get("filetype")) # type: ignore
if filetype_supported
else None
)
Expand Down
20 changes: 20 additions & 0 deletions python-sdk/tests/databases/test_base_database.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,26 @@ def test_create_table_using_columns_raises_exception():
assert exc_info.match("To use this method, table.columns must be defined")


def test_check_schema_autodetection_is_supported():
"""
Test the condition native schema autodetection for files and prefixes
"""
db = create_database("gcp_conn")
assert db.check_schema_autodetection_is_supported(
source_file=File(path="gs://bucket/prefix", filetype=FileType.CSV)
)

assert db.check_schema_autodetection_is_supported(
source_file=File(path="gs://bucket/prefix/key.csv")
)

assert not (
db.check_schema_autodetection_is_supported(
source_file=File(path="s3://bucket/prefix/key.csv")
)
)


def test_subclass_missing_append_table_raises_exception():
db = DatabaseSubclass(conn_id="fake_conn_id")
source_table = Table()
Expand Down

0 comments on commit a2dc914

Please sign in to comment.