Skip to content

v3.38.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 06 Oct 15:56
· 1556 commits to master since this release
v3.38.0
This tag was signed with the committer’s verified signature.
laughingman7743
GPG key ID: A3CA5C588776F5BF
Verified
Learn about vigilant mode.
7fc7f07
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

v3.38.0 is a minor release of the 3.x maintenance line. It backports bug fixes from master (#1104, #903) and contains no breaking changes. The 3.x line still supports Python 3.10 and sqlalchemy>=1.0.0, and its dependency requirements are unchanged.

It includes the fixes for the issues @aminghadersohi reported, #854 and #857, as well as @aminghadersohi's own fix #839. Thank you for the reports and the contribution.

SQLAlchemy

  • Reflected top-level map<...> columns are AthenaMap(key_type, value_type) instead of String. Top-level struct<...> / row(...) columns are AthenaStruct with their field types (#995, #857). A field, key, or value type that the dialect does not recognize is reflected as NullType with a warning.
  • A bare awsathena:// URL resolves to the REST dialect, the same as awsathena+rest://. Before, engine.driver raised AttributeError (#839).
  • to_sql finds an existing table whatever the case of its name (#800, #799).

Cursors

  • TIMESTAMP WITH TIME ZONE values with a numeric UTC offset such as +05:30 convert to aware datetimes with a fixed offset instead of naive ones (#1033, #854). Values with a zone name are unchanged.
  • A synchronous for loop over an asyncio cursor or result set raises TypeError instead of never ending. Use async for (#899, #898).
  • The client-side result cache (cache_size / cache_expiration_time) is no longer used for qmark queries with parameters. Before, it could return the result of an execution that ran with different parameters, because Athena does not return the parameters of earlier executions. A repeated qmark query with the same parameters is now executed again (#959, #941).
  • PandasCursor (#950, #924):
    • Without chunksize, as_pandas() returns the whole result when auto_optimize_chunksize chose a chunk size. It used to return only the first chunk.
    • A second as_pandas() call, or one made after rows were fetched, returns an empty DataFrame instead of raising StopIteration.
    • PandasDataFrameIterator.as_pandas(), as returned by iter_chunks() or by as_pandas() with an explicit chunksize, keeps the chunks' index and categorical dtypes. With index_col, the joined DataFrame is indexed by that column instead of a new RangeIndex.
  • With engine="pyarrow", PandasCursor reads the tab-separated .txt results of DDL statements such as SHOW TABLES and DESCRIBE with the C engine, keeping leading zeros, exponent notation, and padding (#1075, #1073).
  • ArrowCursor:
    • The NULL rows of single-column CSV results are kept (#1031, #1026).
    • Multi-line CSV values that cross a read block are read correctly (#1045, #1040).
    • Arrow decimal256 columns, for example in UNLOAD results, are described as decimal with their precision and scale instead of string (#900, #888).
  • ArrowCursor with managed query result storage (no output location): the fetch methods return the values that as_arrow() holds. TIME, VARBINARY, and JSON values used to raise TypeError because the values were converted twice. A custom Arrow converter mapping is therefore not applied on this path, the same as for as_arrow() (#1023, #1021).
  • A closed PolarsDataFrameIterator stops for every reader kind (#1029, #1019).

Spark

  • Session readiness polling stops with OperationalError when the session is terminated, degraded, or failed, instead of polling forever (#831, #792).
  • A Spark session that a cursor starts is terminated when the cursor setup fails (#832, #794).
  • AsyncSparkCursor.close() shuts down its executor even when session termination fails (#830, #795).

S3 filesystem

  • Appending to an object keeps its existing data. This fixes data loss when appending within a larger block size, truncation by an empty append, and duplicated bytes (#929, #921).
  • A single large write() no longer makes the multipart upload fail with EntityTooSmall (#947, #942).
  • Multipart copies, including appends to objects over 5 GiB, keep their parts within the S3 part size limits (#955, #951).
  • With a short last part, the multipart part boundaries change, so the multipart ETag of identical content can differ from before (#947, #955).
  • Stale listing and object caches are invalidated (#928, #922).

Development

  • The Test workflow runs the SQLAlchemy and Spark suites when shared core modules change, and the PyAthena suite only when its files change (#906, #913). The compliance-suite dialect registry matches the entry points (#905). Cache-hit tests run in the test work group (#817).

What's Changed

New Contributors

Full Changelog: v3.37.0...v3.38.0

Contributors

laughingman7743 and aminghadersohi
Assets 4
Loading