Skip to content

v4.2.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 19 Oct 13:40
· 34 commits to master since this release
v4.2.0
This tag was signed with the committer’s verified signature.
laughingman7743
GPG key ID: A3CA5C588776F5BF
Verified
Learn about vigilant mode.
baee716
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Breaking changes

  • A closed cursor raises ProgrammingError("Cursor is closed.") from execute(), executemany(), the fetch methods, iteration, cancel(), the metadata methods, and its other methods (#1274). Before, fetches returned None or [] and execute() ran a new query. Cursors have a new is_closed property.
  • A closed result set raises ProgrammingError from its fetch methods, iteration, as_pandas(), as_arrow(), as_polars(), and iter_chunks(), instead of returning None, [], or an empty DataFrame or table (#1274).
  • next() on an asyncio cursor or result set raises TypeError instead of returning an un-awaited coroutine (#1274).
  • After a fetch raises, later fetchone(), fetchmany(), and fetchall() calls raise ProgrammingError (#1238). Before, the pandas and Polars result sets returned no more rows, and the other result sets continued after the skipped rows. Invalid arguments, such as a non-integer fetchmany() size, do not count as a failed fetch.
  • Calling one cursor or result set from several threads is not supported, as threadsafety = 2 declares (#1238). This includes cancel() from a timer thread and close() during a fetch. To stop a query from another thread, use its query ID.
  • A cancelled fetch of AioPandasCursor, AioArrowCursor, AioPolarsCursor, or AioS3FSCursor drops the result set from the cursor, so later fetches raise ProgrammingError("No result set.") (#1238). A cancelled as_*() call of these cursors does the same (#1286).
  • AioPandasCursor.as_pandas(), AioPolarsCursor.as_polars() / as_arrow(), and AioArrowCursor.as_arrow() / as_polars() are coroutines: use await cursor.as_pandas() (#1286). AioPandasCursor.as_pandas() always returns a DataFrame. To read chunks one at a time, use the new async iter_chunks() of AioPandasCursor and AioPolarsCursor.
  • After close() of AsyncCursor or one of its variants (dict, pandas, Arrow, Polars, S3FS), the futures of execute(), description(), and poll() that still wait for their query raise ProgrammingError (#1279, #1300). The query is not cancelled. A result set built after the close is closed before its future resolves. The futures of cancel() and query_execution() still run. A result set whose future resolved before the close stays open and belongs to the caller.
  • AsyncCursor.description(query_id) reads only the result set metadata, or the Parquet schema of an UNLOAD result, and no longer reads the result rows (#1300). It uses the new metadata reader classes, such as AthenaResultSetMetadataReader. A custom result set class that overrides the description property no longer changes what description(query_id) returns; make that change in its metadata reader class instead.
  • PandasCursor's temporal dtypes now depend only on the Athena type (#1273). The same applies to AsyncPandasCursor and AioPandasCursor.
    • timestamp is always datetime64[us], and digits beyond microseconds are truncated. It was [ns] for 7 or more fractional digits, [s] when all values were NULL or with engine="pyarrow" and no fraction, [ns] with engine="pyarrow" and a fraction, and str when a column also held values outside the nanosecond range.
    • timestamp with time zone is datetime64[us, UTC]. It had the data's zone, or was object for several zones or an all-NULL column.
    • date from UNLOAD results is datetime64[us], as from CSV results. It was object (date).
    • The fetch methods return datetime, date, and aware datetime in UTC, and None for NULL. They returned pd.Timestamp and NaT.
    • When parse_dates, dtype, or converters is given to execute(), PyAthena leaves the temporal columns to pandas, and the fetch methods return pandas' values for them.
    • engine="pyarrow" is now also used for results with timestamp with time zone columns.
  • A timestamp with time zone value raises DataError when its zone name is unknown, or when its zone repeats or skips its local time (daylight saving time) (#1224). Before, it returned a naive datetime or the earlier instant. Convert such values to UTC in SQL (AT TIME ZONE 'UTC') or read to_unixtime(). tzdata>=2024.2 is required.
  • S3File.multipart_upload is read-only, and S3File.multipart_upload_parts is removed (#1232). S3File.completed_parts returns the uploaded or copied parts in part-number order. commit() raises S3MultipartStateError after discard() or after a failed write or commit. Before, it uploaded the buffered data after a discard and did nothing after a failure.
  • A failed read of an UNLOAD data manifest, or of a Spark calculation's output file through get_std_out() / get_std_error(), raises OperationalError instead of the botocore exception (#1278). Callers that catch botocore errors there must catch OperationalError.
  • Custom result set subclasses: fetchmany() and fetchall() now take rows through _take_rows(). A subclass whose fetchone() does not use the row buffer must override _take_rows() as well (#1195). Several private result set helpers moved to the metadata reader classes or were removed (#1300).

Behavior changes

  • PandasCursor, ArrowCursor, and PolarsCursor read UNLOAD timestamps at microseconds at every nesting level, in whole and chunked reads (#1233). The dtypes are datetime64[us], timestamp[us], and Datetime("us"), as for CSV results. Values outside 1677-09-21 to 2262-04-11 are now correct; before, they were wrapped or saturated.
    • PandasCursor: coerce_int96_timestamp_unit given to execute() still replaces the unit. coerce_int96_timestamp_unit=None is the same as not giving it (#1291).
    • PolarsCursor: a schema given to execute() is used as given.
    • PolarsCursor with Polars 1.x reads UNLOAD timestamps at nanoseconds and casts them to microseconds, so values outside 1677-09-21 to 2262-04-11 are wrong. It logs a warning that recommends Polars 2.0 or later. Chunked reads with pyarrow installed and no read options given to execute() read them correctly (#1282).
  • ArrowCursor reads date from CSV results as date32 instead of timestamp[ms], as from UNLOAD results (#1236). as_polars() gives pl.Date instead of pl.Datetime("ms").
  • ArrowCursor reads NULL time, time with time zone, and timestamp with time zone values of CSV results as null instead of '' (#1236). The fetch methods still return None.
  • In every cursor, a value that fails to convert raises DataError naming the column (Column 'name': ...) (#1273, #1277). PandasCursor whole reads raise it instead of an OperationalError that wrapped it. A DataError from a function given in converters is still wrapped in OperationalError.
  • PolarsCursor with ignore_errors=True truncates a CSV timestamp with more than nine fractional digits instead of returning null (#1234). Text that is not a timestamp is still null.
  • PolarsCursor with Polars 2.0 (#1169):
    • When new_columns is given together with schema_overrides, it must name every column.
    • A map column of an UNLOAD result is described as map, and its values are dicts.
    • ArrowCursor.as_polars() returns map columns as pl.Map.
    • On every Polars version, a struct column that contains a list is described as row instead of array.
  • For a chunked CSV result of PolarsCursor, AioPolarsCursor, and AsyncPolarsCursor, the HeadObject size check runs when the result set is created, and its error is raised there instead of at the first fetch (#1190).
  • The chunk iterators of the pandas and Polars result sets, from iter_chunks() or as_pandas(), raise ProgrammingError at their next read once their result set is closed, for example by another query on the cursor (#1286). Before, they ended without an error.
  • An UNLOAD result of PandasCursor and PolarsCursor that has data files is described by its columns, as in ArrowCursor (#1300). Before, a result whose read options dropped every row was described as []. For a chunked pandas read whose files hold no rows, engine="c" now raises ProgrammingError.
  • to_sql() with the default flavor="spark" creates tables whose column names have spaces, tabs, newlines, and ,;{}()= replaced with _, matching the names in the Parquet files (#1299). Such columns used to read as NULL. Tables created before keep the old names, and appending does not change them; recreate them, for example with if_exists="replace".
  • Cancelling an asyncio call stops the retries of its Athena requests in the thread: status polling, metadata, AioCursor pages, and object construction (#1223). A request already sent still runs to its end. The cleanup requests (StopQueryExecution, StopCalculationExecution, TerminateSession) keep their retries.
  • Cancelling an AioS3FileSystem coroutine stops the retries of its S3 requests (#1231). Cancelling a chunked UNLOAD read of AioPandasCursor or AioPolarsCursor stops the retries of the requests that open its Parquet files (#1297). Closing a chunked UNLOAD result no longer waits for the retries of a Parquet file being opened ahead (#1297).
  • AioS3FileSystem runs its blocking calls, including multipart copies, in a pool of up to max_workers threads of the filesystem (#1184). Each of its files uses a pool of up to max_workers threads. The event loop's default executor no longer limits them, so a process can use more threads than before. A max_workers that is not positive raises ValueError when the filesystem is created. After S3AioExecutor.shutdown(), submit() raises RuntimeError.
  • S3File multipart writes keep at most max_workers parts in flight, counting the part copies of an append, and a write waits until a part finishes (#1165, #1232). A part that failed is raised by the next write(), flush(), or close(), which closes the file without committing it (#1176, #1183, #1232). Before, it was raised only by close() or commit(). Inside an fsspec transaction, the error is raised by the write. If the caller catches it, the transaction fails when it commits. Committing again after a successful commit sends no request.
  • A failed part, completion, or part copy stops the retries of the running requests before the abort, and the requests that have not started are cancelled (#1192, #1231). A failed range of a parallel read cancels the ranges that have not started. A recursive chmod stops after the first failed update, and the keys it has not updated keep their ACL (#1231).
  • In the traceback of an error from an S3 request or from a call run by an S3Executor, the frames below S3Core.call() or the executor no longer hold local variables, for example in pdb.pm() (#1192).
  • executemany() closes only the result set after a failure, not the cursor (#1274).

Benchmark

The benchmark of this release ran on commit 9e99caa8 (#1288), whose package code is identical to this release: the release commit baee716c differs from it only in one test file (#1311). Its history entries are benchmarks/history/2026-10-19-9e99caa8.md and benchmarks/history/2026-10-19-9e99caa8-filesystem.md (#1310). Highlights:

  • pandas.read_sql_query() through the result-file drivers is faster, because the row output of PandasCursor and PolarsCursor is faster (#1167, #1177). For 10 million flat rows, awsathena+pandas with unload=true took 50.7 seconds instead of 162.6, and awsathena+polars with unload=true 48.7 seconds instead of 92.0.
  • AioS3FileSystem writes and copies as fast as S3FileSystem with 40 workers (#1184): a 6 GiB multipart copy ran at 1,889–1,925 MiB/s instead of 521–540.
  • Chunked reads of 100 million flat rows, about 29 GB of CSV, peaked at 462 MiB (PandasCursor CSV), 1.0 GiB (PolarsCursor CSV and UNLOAD), and 986 MiB (PandasCursor UNLOAD), with chunksize=100000.

Known issues

  • PolarsCursor with Polars 1.x holds a copy of the CSV result file while it reads a CSV result whole, so the peak memory of such reads grew by about the size of the file (#1307). Polars 2.0, chunked reads, and UNLOAD results are not affected.

What's Changed

New Contributors

Full Changelog: v4.1.0...v4.2.0

Contributors

laughingman7743 and greg-om
Assets 4
Loading