a possible stopgap solution to issue #531 #532

AJM10565 · 2024-03-29T19:40:21Z

Hopefully this is a decent stopgap measure to the issue I raised.

laughingman7743 · 2024-03-30T04:03:24Z

Thank you, that's a good improvement, could you add a test for the case of a value error?
https://github.com/laughingman7743/PyAthena/blob/master/tests/pyathena/pandas/test_util.py

AJM10565 · 2024-04-01T19:03:06Z

I added some unit tests and also another use case I encountered with a similar issue.

pyathena/pandas/util.py

laughingman7743 · 2024-04-02T15:55:57Z

tests/pyathena/pandas/test_util.py

+            compression="snappy",
+            partitions=["partition_key"],
+        )
+    assert str(exc_info.value) == "Partition key: `partition_key` contains None values, no data will be written to the table."


Please add a new line at the end of the file.

laughingman7743 · 2024-04-02T16:04:41Z

The test you added looks good. Thank you.
If you fix the lint error, I will merge this PR.

$ make test
hatch run chk
cmd [1] | ruff check .
cmd [2] | ruff format --check .
43 files already formatted
cmd [3] | mypy .
Success: no issues found in 41 source files
hatch run test
============================================================================================================================================================================================================================== test session starts ===============================================================================================================================================================================================================================
platform darwin -- Python 3.11.8, pytest-8.0.0, pluggy-1.4.0
rootdir: /Users/foobar/github/PyAthena
configfile: pyproject.toml
plugins: cov-4.1.0, dependency-0.6.0, xdist-3.5.0, anyio-4.2.0
8 workers [502 items]   
.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................... [ 91%]
...................................s........                                                                                                                                                                                                                                                                                                                                                                                                                               [100%]
================================================================================================================================================================================================================================ warnings summary ================================================================================================================================================================================================================================
tests/pyathena/pandas/test_cursor.py::TestPandasCursor::test_complex[pandas_cursor2-1000000]
tests/pyathena/pandas/test_cursor.py::TestPandasCursor::test_complex[pandas_cursor0-None]
tests/pyathena/pandas/test_cursor.py::TestPandasCursor::test_complex[pandas_cursor1-1000]
tests/pyathena/pandas/test_cursor.py::TestPandasCursor::test_complex_as_pandas[pandas_cursor0-None]
tests/pyathena/pandas/test_cursor.py::TestPandasCursor::test_complex_as_pandas[pandas_cursor1-1000]
tests/pyathena/pandas/test_cursor.py::TestPandasCursor::test_complex_as_pandas[pandas_cursor2-1000000]
  /Users/foobar/github/PyAthena/pyathena/pandas/result_set.py:214: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[datetime.time(0, 0)]' has dtype incompatible with datetime64[ns], please explicitly cast to a compatible dtype first.
    df.loc[:, times] = df.loc[:, times].apply(lambda r: r.dt.time)

tests/pyathena/sqlalchemy/test_base.py: 65 warnings
  /Users/foobar/github/PyAthena/tests/pyathena/conftest.py:96: SADeprecationWarning: The dbapi() classmethod on dialect classes has been renamed to import_dbapi().  Implement an import_dbapi() classmethod directly on class <class 'pyathena.sqlalchemy.rest.AthenaRestDialect'> to remove this warning; the old .dbapi() classmethod may be maintained for backwards compatibility.
    return sqlalchemy.engine.create_engine(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

---------- coverage: platform darwin, python 3.11.8-final-0 ----------
Name                                  Stmts   Miss  Cover
---------------------------------------------------------
pyathena/__init__.py                     43      9    79%
pyathena/arrow/__init__.py                0      0   100%
pyathena/arrow/async_cursor.py           42      0   100%
pyathena/arrow/converter.py              32      2    94%
pyathena/arrow/cursor.py                 90      1    99%
pyathena/arrow/result_set.py            148     14    91%
pyathena/arrow/util.py                   45      3    93%
pyathena/async_cursor.py                 54      1    98%
pyathena/common.py                      291     47    84%
pyathena/connection.py                  127     32    75%
pyathena/converter.py                    81      9    89%
pyathena/cursor.py                       72      1    99%
pyathena/error.py                        21      0   100%
pyathena/fastparquet/__init__.py          0      0   100%
pyathena/fastparquet/util.py             44      3    93%
pyathena/filesystem/__init__.py           0      0   100%
pyathena/filesystem/s3.py               279     74    73%
pyathena/filesystem/s3_object.py         34      0   100%
pyathena/formatter.py                    99      5    95%
pyathena/model.py                       490     13    97%
pyathena/pandas/__init__.py               3      0   100%
pyathena/pandas/async_cursor.py          44      0   100%
pyathena/pandas/converter.py             23      0   100%
pyathena/pandas/cursor.py                97      1    99%
pyathena/pandas/result_set.py           230     26    89%
pyathena/pandas/util.py                 160      6    96%
pyathena/result_set.py                  523     97    81%
pyathena/spark/__init__.py                0      0   100%
pyathena/spark/async_cursor.py           34      5    85%
pyathena/spark/common.py                188     48    74%
pyathena/spark/cursor.py                 32      2    94%
pyathena/sqlalchemy/__init__.py           0      0   100%
pyathena/sqlalchemy/arrow.py             15     15     0%
pyathena/sqlalchemy/base.py             526     71    87%
pyathena/sqlalchemy/pandas.py            19     19     0%
pyathena/sqlalchemy/requirements.py      98     98     0%
pyathena/sqlalchemy/rest.py               4      0   100%
pyathena/sqlalchemy/types.py             31      9    71%
pyathena/sqlalchemy/util.py               3      1    67%
pyathena/util.py                         31      1    97%
---------------------------------------------------------
TOTAL                                  4053    613    85%
Coverage HTML written to dir htmlcov

============================================================================================================================================================================================================ 501 passed, 1 skipped, 71 warnings in 329.07s (0:05:29) =============================================================================================================================================================================================================

Co-authored-by: laughingman7743 <laughingman7743@gmail.com>

AJM10565 · 2024-04-02T20:33:19Z

when I run
make test hatch run chk cmd [1] | ruff check . cmd [2] | ruff format --check . 43 files already formatted cmd [3] | mypy . Success: no issues found in 41 source files hatch run test ImportError while loading conftest '/Users/allanmiller/Source/PyAthena/tests/pyathena/conftest.py'. tests/__init__.py:38: in <module> ENV = Env() tests/__init__.py:15: in __init__ assert self.region_name, "Required environment variable AWS_DEFAULT_REGIONnot found." E AssertionError: Required environment variableAWS_DEFAULT_REGIONnot found. make: *** [test] Error 4
So I'm not certain if I missed something. Please let me know if I did.

laughingman7743 · 2024-04-03T02:57:45Z

For more information on running the test, please check the following
https://laughingman7743.github.io/PyAthena/testing.html
Some environment variables are required; an AWS account is also required.

laughingman7743 · 2024-04-03T03:02:57Z

$ make test
hatch run chk
cmd [1] | ruff check .
cmd [2] | ruff format --check .
43 files already formatted
cmd [3] | mypy .
Success: no issues found in 41 source files
hatch run test
============================================================================================================================================================================================================================== test session starts ===============================================================================================================================================================================================================================
platform darwin -- Python 3.11.8, pytest-8.0.0, pluggy-1.4.0
rootdir: /Users/foobar/github/PyAthena
configfile: pyproject.toml
plugins: cov-4.1.0, dependency-0.6.0, xdist-3.5.0, anyio-4.2.0
8 workers [502 items]   
...........................................................................................................................................................................................................................................................................................................................................................................................................................................................s.............. [ 91%]
............................................                                                                                                                                                                                                                                                                                                                                                                                                                               [100%]
================================================================================================================================================================================================================================ warnings summary ================================================================================================================================================================================================================================
tests/pyathena/pandas/test_cursor.py::TestPandasCursor::test_complex[pandas_cursor0-None]
tests/pyathena/pandas/test_cursor.py::TestPandasCursor::test_complex_as_pandas[pandas_cursor0-None]
tests/pyathena/pandas/test_cursor.py::TestPandasCursor::test_complex[pandas_cursor1-1000]
tests/pyathena/pandas/test_cursor.py::TestPandasCursor::test_complex_as_pandas[pandas_cursor1-1000]
tests/pyathena/pandas/test_cursor.py::TestPandasCursor::test_complex[pandas_cursor2-1000000]
tests/pyathena/pandas/test_cursor.py::TestPandasCursor::test_complex_as_pandas[pandas_cursor2-1000000]
  /Users/foobar/github/PyAthena/pyathena/pandas/result_set.py:214: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[datetime.time(0, 0)]' has dtype incompatible with datetime64[ns], please explicitly cast to a compatible dtype first.
    df.loc[:, times] = df.loc[:, times].apply(lambda r: r.dt.time)

tests/pyathena/sqlalchemy/test_base.py: 65 warnings
  /Users/foobar/github/PyAthena/tests/pyathena/conftest.py:96: SADeprecationWarning: The dbapi() classmethod on dialect classes has been renamed to import_dbapi().  Implement an import_dbapi() classmethod directly on class <class 'pyathena.sqlalchemy.rest.AthenaRestDialect'> to remove this warning; the old .dbapi() classmethod may be maintained for backwards compatibility.
    return sqlalchemy.engine.create_engine(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

---------- coverage: platform darwin, python 3.11.8-final-0 ----------
Name                                  Stmts   Miss  Cover
---------------------------------------------------------
pyathena/__init__.py                     43      9    79%
pyathena/arrow/__init__.py                0      0   100%
pyathena/arrow/async_cursor.py           42      0   100%
pyathena/arrow/converter.py              32      2    94%
pyathena/arrow/cursor.py                 90      1    99%
pyathena/arrow/result_set.py            148     14    91%
pyathena/arrow/util.py                   45      3    93%
pyathena/async_cursor.py                 54      1    98%
pyathena/common.py                      291     47    84%
pyathena/connection.py                  127     32    75%
pyathena/converter.py                    81      9    89%
pyathena/cursor.py                       72      1    99%
pyathena/error.py                        21      0   100%
pyathena/fastparquet/__init__.py          0      0   100%
pyathena/fastparquet/util.py             44      3    93%
pyathena/filesystem/__init__.py           0      0   100%
pyathena/filesystem/s3.py               279     74    73%
pyathena/filesystem/s3_object.py         34      0   100%
pyathena/formatter.py                    99      5    95%
pyathena/model.py                       490     13    97%
pyathena/pandas/__init__.py               3      0   100%
pyathena/pandas/async_cursor.py          44      0   100%
pyathena/pandas/converter.py             23      0   100%
pyathena/pandas/cursor.py                97      1    99%
pyathena/pandas/result_set.py           230     26    89%
pyathena/pandas/util.py                 160      6    96%
pyathena/result_set.py                  523     97    81%
pyathena/spark/__init__.py                0      0   100%
pyathena/spark/async_cursor.py           34      5    85%
pyathena/spark/common.py                188     48    74%
pyathena/spark/cursor.py                 32      2    94%
pyathena/sqlalchemy/__init__.py           0      0   100%
pyathena/sqlalchemy/arrow.py             15     15     0%
pyathena/sqlalchemy/base.py             526     71    87%
pyathena/sqlalchemy/pandas.py            19     19     0%
pyathena/sqlalchemy/requirements.py      98     98     0%
pyathena/sqlalchemy/rest.py               4      0   100%
pyathena/sqlalchemy/types.py             31      9    71%
pyathena/sqlalchemy/util.py               3      1    67%
pyathena/util.py                         31      1    97%
---------------------------------------------------------
TOTAL                                  4053    613    85%
Coverage HTML written to dir htmlcov

============================================================================================================================================================================================================ 501 passed, 1 skipped, 71 warnings in 328.65s (0:05:28) =============================================================================================================================================================================================================

a possible stopgap solution to issue laughingman7743#531

71e46e6

laughingman7743 linked an issue Mar 30, 2024 that may be closed by this pull request

If a value for a partition key is None, to_sql doesn't warn you and no data is written #531

Closed

added another possibility and unit tests

b049a6f

laughingman7743 reviewed Apr 2, 2024

View reviewed changes

pyathena/pandas/util.py Outdated Show resolved Hide resolved

laughingman7743 reviewed Apr 2, 2024

View reviewed changes

AJM10565 and others added 2 commits April 2, 2024 15:29

Update pyathena/pandas/util.py

d08d8d4

Co-authored-by: laughingman7743 <laughingman7743@gmail.com>

added new line to end of file

ad78ec1

laughingman7743 merged commit e808009 into laughingman7743:master Apr 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

a possible stopgap solution to issue #531 #532

a possible stopgap solution to issue #531 #532

AJM10565 commented Mar 29, 2024

laughingman7743 commented Mar 30, 2024

AJM10565 commented Apr 1, 2024

laughingman7743 Apr 2, 2024

AJM10565 Apr 2, 2024

laughingman7743 commented Apr 2, 2024

AJM10565 commented Apr 2, 2024

laughingman7743 commented Apr 3, 2024

laughingman7743 commented Apr 3, 2024

a possible stopgap solution to issue #531 #532

a possible stopgap solution to issue #531 #532

Conversation

AJM10565 commented Mar 29, 2024

laughingman7743 commented Mar 30, 2024

AJM10565 commented Apr 1, 2024

laughingman7743 Apr 2, 2024

Choose a reason for hiding this comment

AJM10565 Apr 2, 2024

Choose a reason for hiding this comment

laughingman7743 commented Apr 2, 2024

AJM10565 commented Apr 2, 2024

laughingman7743 commented Apr 3, 2024

laughingman7743 commented Apr 3, 2024