Skip to content

v4.0.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 11 Oct 02:49
· 579 commits to master since this release
v4.0.0
This tag was signed with the committer’s verified signature.
laughingman7743
GPG key ID: A3CA5C588776F5BF
Verified
Learn about vigilant mode.
55dda52
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

v4.0.0 changes the project's copyright notices. Please read the Copyright and license notices section first and review NOTICE. The license itself is still the MIT License.

v4.0.0 is a major release from master, which was branched from the 3.x line at v3.36.0. It drops Python 3.10, raises the minimum versions of the core and optional dependencies, makes connection and cursor settings keyword-only, and rejects arguments that used to be ignored silently. It also contains a large number of fixes and behavior changes across the cursors, the SQLAlchemy dialect, Spark, and the S3 filesystem.

Read Breaking changes before upgrading. The usage guide has migration sections for the keyword-only arguments, the worker settings, and the environment fallbacks ("... in PyAthena 4.0" in docs/usage.md), and the Spark and S3FS guides have their own 4.0 sections. Users who need Python 3.10, pandas 2, an older pyarrow or Polars, or SQLAlchemy 1.x can stay on the 3.x maintenance line (latest v3.39.0).

Several fixes in this release were already backported to 3.x in v3.37.0, v3.38.0, and v3.39.0. They are listed under Already in 3.x releases at the end and marked in "What's Changed"; users upgrading from v3.39.0 already have them.

Thanks to @mvanhorn for the STRUCT DDL fix (#870), to @aminghadersohi for the bare awsathena:// scheme fix (#839) and for reporting #854, #855, and #857, and to @innicoder for reporting the memory issue #417, which #1001 addresses.

Copyright and license notices

  • The copyright holder in LICENSE changes from "laughingman7743" to "The PyAthena authors" (Copyright 2017 The PyAthena authors). The license terms are unchanged (MIT) (#809, #790).
  • The package metadata lists "The PyAthena authors" as the author and laughingman7743 as the maintainer, and uses PEP 639 license metadata: the license expression MIT with LICENSE and NOTICE as license files, so both files are included in the sdist and wheel (#809).
  • Files that contain only original work by the PyAthena authors carry the header Copyright <year> The PyAthena authors with SPDX-License-Identifier: MIT. Existing files written solely by the maintainer received it, new original files must carry it, and a check enforces the headers across the repository (#805, #808).
  • The new NOTICE file acknowledges PyHive, which informed PyAthena's early development, and lists the parts that still follow it, with PyHive's Apache License 2.0 notice. It also records the other third-party material (the GitHub Actions OIDC CloudFormation template and strtobool) and explains which files do not carry the header, such as files with contributions from other contributors, whose authorship is recorded in the Git history (#802). The per-file classification is recorded in #790.
  • Redistributors and packagers should include NOTICE together with LICENSE.

Breaking changes

Python and dependencies

  • Python 3.10 is no longer supported; PyAthena requires Python 3.11 or later (#884, #864).
  • Minimum versions of the core dependencies (#1036, #1055, #1126, #1125):
    • boto3>=1.43.31 and botocore>=1.43.31 (was 1.41.2). botocore 1.43.31 is the first release with the S3 annotation operations used by multipart copies.
    • fsspec>=2026.9.0 (no lower bound before).
    • tenacity>=8.0.1 (was 4.1.0). Older releases break retry_api_call(): releases before 6.1.0 fail on Python 3.11+, and 8.0.0 raises TypeError from after_log on the first retry. (Also in v3.39.0.)
    • python-dateutil is no longer a dependency; tzdata is required.
  • Minimum versions of the extras (#858, #885, #828):
    • sqlalchemy: sqlalchemy>=2.0.0 (was 1.0.0). Environments pinned to SQLAlchemy 1.4 cannot install pyathena[sqlalchemy] 4.0.0.
    • pandas: pandas>=3.0.0.
    • arrow: pyarrow>=22.0.0.
    • polars: polars>=1.39.0 (was 1.0.0). This applies even to users who never set chunksize: Polars 1.34–1.36 silently truncate chunked reads on errors, and Polars before 1.39 fill a local file cache that is not released.
  • The package is marked Development Status :: 5 - Production/Stable (#904, #865).

Connections and cursors

  • Connection settings and cursor constructor settings are keyword-only; positional settings raise TypeError. execute() keeps operation and parameters positional and requires keywords after them. connection.cursor(CursorClass) still works (#1101, #1092).
  • Unknown cursor constructor keywords, given directly or through the connection's cursor_kwargs, raise TypeError. execute() of the default, dict, Arrow, S3FS, and Spark cursors raises TypeError for unknown keywords before changing cursor state or starting a query. The pandas and Polars cursors still pass unused execute() keywords to the reader, which rejects unknown ones when it reads the results. connection.cursor(OtherCursor) raises if cursor_kwargs contain options that OtherCursor does not accept. A cursor subclass that passes its own settings from cursor_kwargs on to the base constructor must consume them first (#1102, #1093).
  • The S3 read worker argument of PandasCursor, PolarsCursor, AioPandasCursor, and AioPolarsCursor is renamed from max_workers to s3_max_workers, in the constructor and in execute(). In AsyncPandasCursor and AsyncPolarsCursor, max_workers sizes only the cursor's thread pool and s3_max_workers sets the S3 read workers; pass both to AsyncPolarsCursor to keep its former values. The old name raises TypeError (in execute(), when the result set is created after the query has run) (#1103, #1094).
  • arraysize passed at cursor creation now takes effect and is validated: Cursor, DictCursor, AioCursor, AioDictCursor, AsyncCursor, and AsyncDictCursor raise ProgrammingError for values above 1000, and every SQL cursor raises it for zero or negative values. Before, the value was ignored by Cursor, DictCursor, AioCursor, and AioDictCursor (since v2.0.0), so a small value now means more GetQueryResults calls. The Arrow, pandas, Polars, and S3FS cursors still accept values above 1000 (#901, #897).
  • connect(endpoint_url=...) no longer applies to S3. Setups that serve Athena and S3 from one endpoint (for example LocalStack) need AWS_ENDPOINT_URL_S3 or AWS_ENDPOINT_URL. This affects the result sets, S3FileSystem(connection=...), Polars storage_options, the Spark cursors, and to_sql() (#1058, #1011).
  • The connection's work_group follows the same environment fallback rule as s3_staging_dir: only an omitted argument or None reads AWS_ATHENA_WORK_GROUP; an explicit empty string skips the fallback and means no work group (#1108, #1096).
  • DefaultCSVReader (S3FS cursor) is renamed to EmptyStringAsNullCSVReader, without an alias (#1107, #1097).
  • Removed compatibility APIs (#1100, #1095, #1088, #1087):
    • pyathena.formatter._escape_presto; use _escape_trino.
    • The individual execution keywords of the private BaseCursor._execute() / AioBaseCursor._execute(); pass options=ExecuteOptions(...).
    • pyathena.util.strtobool() returns bool instead of int; call int(strtobool(value)) where an integer is required.

SQLAlchemy

  • Resolve the bare awsathena scheme to the REST dialect by @aminghadersohi in #839 (also in v3.38.0)
  • pyathena.sqlalchemy.types.get_double_type is removed; use sqlalchemy.types.DOUBLE (#860).
  • AthenaDialect declares SQLAlchemy 2.0's type_compiler_cls instead of the legacy type_compiler, so code that reads AthenaDialect.type_compiler on the class no longer finds the compiler class (#895, #894).
  • Type rendering is split into a Hive DDL compiler and a Trino DML compiler (#961):
    • TypeEngine.compile(dialect) renders AthenaStruct as STRUCT<name:type> instead of ROW(name type), and Integer (including subclasses and TypeDecorators over it) as INT in all DDL.
    • Integer().with_variant(OtherType(), "awsathena") renders the variant in CREATE TABLE.
    • A JSON column, an AthenaStruct without fields, and NullType in DDL raise CompileError instead of emitting DDL that Athena rejects. CAST(... AS JSON) is unchanged. CLOB/NCLOB render STRING instead of BINARY.

Spark

  • SparkCursor, AsyncSparkCursor, and AioSparkCursor raise NotSupportedError for any parameters value other than None, and their execute() no longer takes work_group (TypeError, even for None). Before, both were silently ignored (#1108, #1096).
  • Closing a Spark cursor created with session_id no longer terminates that session. Pass terminate_session_on_close=True to keep the previous behavior (#844, #793, #796).

S3 filesystem

  • S3FileSystem.PATTERN_PATH and S3FileSystem.parse_path() / AioS3FileSystem.parse_path() are removed. Use S3Path.parse(path) and its bucket, key, and version_id attributes (#1100, #1052).
  • S3FileSystem.OBJECT_ACLS, BUCKET_ACLS, and the MULTIPART_UPLOAD_* limits are removed; use S3Core.OBJECT_ACLS / S3Core.BUCKET_ACLS (for example fs.core.OBJECT_ACLS) (#1090).
  • The max_worker keyword of open() and pipe_file() is removed; pass max_workers (#948, #925).
  • open() for writing or appending (wb, ab, xb) raises ValueError for a path with ?versionId= (#958, #936).
  • find(maxdepth=n) lists n levels instead of n + 1, and maxdepth=0 raises ValueError. rm(), glob(), du(), expand_path(), copy(), and get() pass maxdepth through find(), so rm(path, recursive=True, maxdepth=1) no longer deletes the objects one level below the path. Callers relying on the old numbering need maxdepth + 1 (#956, #933).
  • S3FileSystem.request_kwargs is a read-only property that returns a copy; pass requester_pays=True to the constructor instead of changing it (#1060).
  • S3File calls submit_to() on its executor. An executor passed to S3File that does not subclass S3Executor must provide it (#1113).

Dependencies and packaging

  • New public API: S3Core (S3FileSystem.core, AioS3FileSystem.core) with typed listing entries, pages, and delete/multipart/copy/tagging/ACL/presign requests; S3Path; S3MultipartWriter; S3Executor.submit_to(); connect(s3_config=...) and Connection.s3_client; TERMINAL_STATES on the query and calculation execution models (#1060, #1052, #1085, #1113, #1058, #881).
  • Every public API in pyathena/ has a Google-style docstring, checked with ruff (#919).

Connections and cursors

  • With the default kill_on_interrupt=True, an interrupt during execute() of a SQL cursor now propagates as KeyboardInterrupt / asyncio.CancelledError after the query is stopped, instead of surfacing as OperationalError or being lost when the query succeeded. asyncio.wait_for() now raises TimeoutError. An interrupt while a query is being started waits for StartQueryExecution and stops the query it started (#853, #840).
  • An explicit cursor() argument now overrides the same key in the connection's cursor_kwargs; before, cursor_kwargs won silently (#872, #859).
  • dict_type passed to one dict cursor no longer changes the row type of other dict cursors, and dict_type=None uses the default (#930, #923).
  • One S3 client and connection pool per connection is shared across result sets and filesystems, and Connection.close() closes the network connections of its clients. SQLAlchemy URLs with use_ssl=false now disable TLS, as connect(use_ssl=False) does (#1058, #1011).
  • Result-set filesystems are kept out of fsspec's instance cache, and the pandas result set reads through one filesystem and owns the CSV file it reads (#1001, #417, #978).
  • Managed query result storage (no output location): the pandas, Arrow, and Polars cursors read the results as an S3 CSV result file would be read, with the same types and values, and the cursor's converter applies to them, including a custom one. For example, pandas date columns are datetime64, array/map/row are Athena text, a result without rows has its columns, and result_set_type_hints no longer applies to these cursors on managed storage (#1061, #1028, #1042).
  • ArrowCursor reads timestamp columns as timestamp[us] instead of timestamp[ms]; the Arrow and (unchunked) Polars cursors read timestamps with more than 6 fractional digits, truncated to the column's unit (#1061).
  • Time zones (#1033, #1018, #1030, #1125):
    • time with time zone values are aware datetime.time objects in Cursor, S3FSCursor, ArrowCursor, PolarsCursor, and PandasCursor (strings or naive times before).
    • timestamp with time zone values are aware datetime objects in the Arrow and Polars cursors and in every PandasCursor column.
    • Zone-name values use zoneinfo.ZoneInfo instead of dateutil.tz.tzfile; instants and offsets are unchanged. With PandasCursor, the dtype changes from, for example, datetime64[us, tzfile('/usr/share/zoneinfo/America/New_York')] to datetime64[us, America/New_York]. A zone that the time zone database cannot provide logs a warning.
    • TIME(0) and TIME(p) with p > 6 no longer raise ValueError.
  • pandas results:
    • Without chunksize, repeated as_pandas() / as_polars() / as_arrow() calls return the whole result, as_pandas() / as_polars() return the same object until close() or the next execute(), and fetches after them return the rows (#1005, #935).
    • JSON columns with NULL keep exact decoded values with None for NULL (object dtype), and integer columns on managed storage are Int64 (#1044, #1027). CSV reads with the C engine and PyAthena's JSON converter default to low_memory=False to avoid DtypeWarning (#1080, #1078).
    • With engine="pyarrow", CSV results are read with pyarrow.csv: multi-line values that cross a read block are read correctly, and string columns are no longer changed ("007" used to become "7.0") (#1057, #1048, #1065, #1062).
    • Columns with the same name keep their own values and types (#1050, #1032, #1066, #1051; also for the Arrow and Polars cursors).
    • execute() can override the cursor's read options, and AsyncPandasCursor(block_size=..., cache_type=..., auto_optimize_chunksize=...) now takes effect (#1022, #938).
    • to_sql() uses the connection's config/s3_config and, with connect(session=...), that session's credentials (#1068, #1067).
  • JSON: PandasCursor and ArrowCursor return None for SQL NULL json values; with managed storage, json columns in as_arrow() / as_polars() are string columns, and S3FSCursor uses the configured converter (#1010, #934).
  • PolarsCursor with unload=True and chunksize describes and returns the Parquet columns (broken since v3.24.0) (#1012, #1007).
  • ArrowCursor maps Arrow decimal32/decimal64 columns to decimal (decimal256 also in v3.38.0) (#900, #888).
  • get_table_metadata(), list_table_metadata(), and list_databases() answer a throttled request from the AWS Glue Data Catalog in AwsDataCatalog and S3 Tables catalogs. The fallback needs glue:GetTable, glue:GetTables, and glue:GetDatabases; without them, the Athena request is retried as before after one failed Glue call and a warning. Turn it off with connect(glue_metadata_fallback=False) or glue_metadata_fallback=false in a SQLAlchemy URL. retry_api_call() / async_retry_api_call() accept a keyword-only stop_on predicate (#803, #786, #890, #887).
  • executemany() reports the total affected row count instead of -1 (#768, #754).

SQLAlchemy

  • executemany inserts are batched into multi-row INSERT statements (insertmanyvalues, 100 rows per statement by default; rowcount stays the total). A failed statement fails one page of rows and earlier pages stay inserted, and before_cursor_execute listeners see multi-row statements. Very wide rows may exceed Athena's query length limit; lower insertmanyvalues_page_size or set use_insertmanyvalues=False. Both options are also accepted in the URL query and by engine_from_config (#838, #835, #843).
  • Datetime literals and precision (#818, #756):
    • Generic Date and DateTime render typed DATE '...' / TIMESTAMP '...' literals, and literal_execute no longer raises NotImplementedError.
    • Timestamp literals keep microseconds (6 digits when the value has a sub-millisecond part, 3 otherwise), and cast() to DateTime renders TIMESTAMP(6). Hive CTAS and UNLOAD (including cursors with unload=True) reject timestamp(6) output columns; use cast(col, AthenaTimestamp(precision=3)) or CAST(... AS TIMESTAMP(3)) there.
    • AthenaTimestamp(precision=p) (0–6) casts to TIMESTAMP(p) and truncates bound values.
  • cast() to a TypeDecorator (or a with_variant() type) casts to its implementation type (#836, #829).
  • Reflection:
    • Top-level map<...> columns reflect as AthenaMap and struct<...> / row(...) columns as AthenaStruct with their fields; unparsable types reflect as NullType with a warning (#995, #886, #857; reflection also in v3.38.0).
    • Throttled metadata requests fall back to information_schema, and listed metadata is reused within one Inspector (#777, #780). The dialect's own queries always use an API cursor, whatever cursor_class is configured, and a federated catalog's missing table is resolved from information_schema (#798, #781, #783, #785).
    • has_table() results are cached until Inspector.clear_cache(), and table names up to 255 characters can be reflected (#771).
    • A missing table in a federated catalog raises NoSuchTableError from get_table_comment() and get_table_options() (#801). An empty Glue comment reflects as None (#782).
  • JSON columns return the value Athena returns: type_coerce(literal_column("CAST('<text>' AS JSON)"), JSON) returns the str instead of decoding it a second time (use json_parse() for the parsed value), JSON objects and arrays no longer raise TypeError, and json_deserializer no longer applies to json results (#1010, #934).
  • CREATE TABLE compression (#1025, #1020, #1015, #937):
    • Parquet, text, and JSON tables created with awsathena_compression now get the requested codec: the DDL emits parquet.compression / write.compression, which Athena applies (it ignored the former properties and used GZIP). to_sql() tables also carry parquet.compression. Fix existing tables with ALTER TABLE ... SET TBLPROPERTIES.
    • file_format is matched case-insensitively.

Spark

  • After kill_on_interrupt cancellation, Spark cursors re-raise KeyboardInterrupt / asyncio.CancelledError instead of returning or raising OperationalError (#833, #797).
  • Spark cursors send a generated ClientRequestToken unless client_request_token is given, and an interrupt while a calculation is being started stops that calculation (#861, #841).

S3 filesystem

The filesystem was audited against fsspec and s3fs and restructured around the typed S3Core. Behavior changes, by area:

  • Paths and versions (#1055, #979, #957, #960, #1024, #1084):
    • Keys containing ? can be used; only a version query such as ?versionId=x is read as a version.
    • ls(versions=True) returns version-qualified names, and open(), cat_file(), and get_file() accept version_id (#958).
    • Each explicit version is cached separately, and deleting a version invalidates the object path.
  • Listings and lookups (#993, #1006, #1002):
    • find(path, withdirs=True) includes path itself when it is a directory, prefix levels count toward maxdepth, and find(object_path, maxdepth=n) returns the object.
    • A cached parent listing answers info(), exists(), isfile(), size(), and open(). Objects written by others after the listing was cached are reported missing until refresh=True or invalidate_cache(), as in s3fs.
    • exists() returns True for a bucket that HeadBucket denies.
  • Reads (#987, #998, #990):
    • cat_file() / cat_ranges() return b"" for empty, reversed, or past-the-end ranges, and a negative start without an end reads a suffix range.
    • open(prefix, "rb") and get_file() of a prefix raise FileNotFoundError.
    • get_file() follows fsspec's contract for directories and file objects.
  • Writes (#992, #968, #1003, #1017, #1009, #1038, #1047):
    • A failed pipe()/put_file() leaves the existing object unchanged and aborts its multipart upload.
    • open(path, "xb"), put_file(mode="create"), and pipe_file(mode="create") raise FileExistsError for an existing object.
    • Writes needing more than 10,000 parts and block sizes above 5 GiB raise ValueError; failed multipart uploads wait for running parts before aborting, and an upload whose abort failed is kept for discard().
    • pipe_file(..., compression=...) works for small values, and an unsupported codec raises ValueError.
  • Copies and moves (#1036, #955, #1013, #993):
    • Copies over 5 GiB keep the source's content headers, metadata, tags, and annotations by default, as CopyObject does. They need s3:GetObjectTagging and the annotation permissions; use TaggingDirective="REPLACE" or AnnotationDirective="EXCLUDE" to skip them.
    • mv() deletes only the copied source objects, moves directory trees, and raises ValueError for conflicting destinations.
  • Request parameters (#1000, #1024, #985):
    • Parameters are sent only to the operations that accept them, and parameters given to open(), pipe(), or put() take precedence over s3_additional_kwargs.
    • Lookup parameters such as ExpectedBucketOwner and SSE-C keys are sent with lookups.
    • The s3fs profile argument is used as the boto3 profile name.
  • rm() raises OSError when S3 cannot delete some objects, and rm("s3://bucket", recursive=True) on AioS3FileSystem (or a bucket path with //) raises ValueError instead of emptying the bucket (#986).
  • setxattr() keeps system metadata, storage class, and encryption, and rejects version paths (#1016, #975).
  • AioS3FileSystem matches S3FileSystem in transactions, open(), and touch() (#988). Interrupted or cancelled multipart creations, part scheduling, and copies abort their uploads (#1056, #1074, #1085, #1113, #1122).
  • S3Object follows the mapping contract: a missing key raises KeyError and an unknown attribute raises AttributeError (#991).
  • Multipart uploads and object versions listed for a key path include only that key and the keys under it (#989).

CI, tests, and development

  • The AWS cost of the integration tests is substantially lower without reducing real-service coverage (#834, #846).
  • The Test workflow runs the PyAthena, SQLAlchemy, and Spark suites in parallel, only for pull requests marked ready and only for the suites whose paths changed, on the newest Python version; fork pull requests skip the AWS jobs (#814, #837, #863, #906, #913, #789).
  • The Release workflow runs every suite on every supported Python version before publishing (#862).
  • Test isolation and stability: each test session gets its own S3 Tables namespace, the executemany and partition tests use their own tables, cache-hit tests run in the test work group, and tests are rerun once after a service-side Athena query failure (#815, #873, #817, #811).
  • Test setup: the shared test tables are defined in Python with generated data, fixtures are set up only in processes that run tests, and SQLAlchemy compliance fixture rows are inserted once for read-only classes (#874, #866, #867).
  • Leaked test databases are swept daily (#779, #907).
  • Test coverage: SQLAlchemy parameter and SQL rendering compliance tests are re-enabled, and the test conventions and regression assertions were audited (#825, #1082).
  • The locked data libraries were upgraded and the code is type-checked per interpreter (#822).
  • New cursor benchmark tooling with a fleet runner (#773, #813, #823, #824, #827).
  • MIT license headers, a license header check, and a collective LICENSE attribution with PEP 639 metadata; NOTICE acknowledges PyHive (#802, #805, #808, #809).

Already in 3.x releases

These changes are included in 4.0.0 and were already released on the 3.x line:

  • v3.37.0: SQLAlchemy ARRAY support (#774, #775, #776), STRUCT DDL (#870), DOUBLE casts (#826), CSV binary NULLs (#770).
  • v3.38.0: bare awsathena:// scheme (#839), to_sql table name case (#800), synchronous iteration of asyncio cursors raises TypeError (#899), qmark cache (#959), as_pandas() with auto chunking (#950), Spark session fixes (#830, #831, #832), ArrowCursor fixes (#1023, #1031, #1045), PolarsDataFrameIterator close (#1029), PandasCursor DDL text (#1075), S3 append and multipart fixes (#928, #929, #947, #955), and parts of #900, #995, and #1033.
  • v3.39.0: Arrow session credentials (#1116), Iceberg detection (#1043), native array and typed JSON fixes (#1054, #1127), parts of #1005 and #1033, and the tenacity floor (#1126).

What's Changed

Dependencies and packaging

Connections, cursors, and result sets

SQLAlchemy

Spark

S3 filesystem

Documentation

CI, tests, and development

New Contributors

Full Changelog: v3.36.0...v4.0.0

Contributors

mvanhorn, laughingman7743, and 4 other contributors
Assets 4
Loading