Skip to content

0.8.0

Latest
Latest

Choose a tag to compare

View all tags
@dongjoon-hyun dongjoon-hyun released this 13 Nov 08:40
0.8.0
8f50143

Apache Spark™ Connect Client for Swift language is a subproject of Apache Spark and aims to provide Swift implementation of Spark Connect. 0.8.0 is the 8th release of Apache Spark Connect for Swift client. This is still experimental.

Website

https://apache.github.io/spark-connect-swift/

Swift Package Index

https://swiftpackageindex.com/apache/spark-connect-swift

Documentation

https://swiftpackageindex.com/apache/spark-connect-swift/0.8.0/documentation/sparkconnect

Full Changelog

0.7.0...0.8.0

Resolved Issues

  • [SPARK-61339] Upgrade gRPC Swift NIO Transport to 2.12.0
  • [SPARK-61338] Use Apache Spark 4.0.5, 4.1.4, and 4.2.1 Docker images in CI
  • [SPARK-61330] Use Apache Spark 4.3.0 official release and Docker image in CI and docs
  • [SPARK-60619] Use Apache Spark 4.0.5, 4.1.4, and 4.2.1 instead of RCs
  • [SPARK-60516] Fix DataFrame.toDF to rename columns instead of selecting them
  • [SPARK-60511] Make SparkSession.Builder.create public
  • [SPARK-60477] Fix DataFrameStatFunctions.approxQuantile to throw instead of crashing on empty results
  • [SPARK-60476] Remove unused ArrowFlight files
  • [SPARK-60472] Fix collect to throw an error instead of misreading compressed Arrow batches
  • [SPARK-60471] Fix DataFrame.withColumnRenamed to not crash on duplicate column names
  • [SPARK-60464] Fix completed executions to not fail when the final ReleaseExecute fails
  • [SPARK-60440] Migrate notify_test_workflow.yml from pull_request_target to workflow_run
  • [SPARK-60435] Enable gRPC keepalive by default and support grpc_keepalive_* parameters in connection string
  • [SPARK-60434] Send client_observed_server_side_session_id to detect server-side session changes
  • [SPARK-60426] Make the index-based subscript of Row public
  • [SPARK-60412] Fix SparkSession.Builder to not send spark.(remote|master) to the server
  • [SPARK-60409] Fix createDataFrame to keep the number of rows of local data without columns
  • [SPARK-60407] Fix command APIs to not return empty results when ExecutionProgress arrives first
  • [SPARK-60405] Fix collect to return empty Rows instead of crashing on results without columns
  • [SPARK-60404] Use lowercase UUIDs as operation IDs to fix ReattachExecute and ReleaseExecute
  • [SPARK-60390] Fix collect to return struct columns as nested Rows instead of crashing on non-UTF-8 binary fields
  • [SPARK-60389] Fix createDataFrame to throw instead of crashing on a non-integer spark.sql.session.localRelationCacheThreshold
  • [SPARK-60388] Fix DataFrame.na and DataFrame.stat.sampleBy to not crash on unsupported value types
  • [SPARK-60352] Fix DataFrameStatFunctions.cov/corr/approxQuantile to throw instead of crashing on unexpected result types
  • [SPARK-60353] Fix Catalog APIs to not crash on null values returned by the server
  • [SPARK-60351] Fix getConf(WithDefault|Option)? and isModifiable to not crash on empty Config responses
  • [SPARK-60349] Fix StreamingQuery creation to throw instead of trapping on non-UUID query ids from the server
  • [SPARK-60303] Support UDFRegistration.registerJava via SparkSession.udf
  • [SPARK-60298] Set default grpc_max_message_size to 128 MiB and support Arrow batch result chunking
  • [SPARK-60299] Fix SparkSession to reject an invalid session_id instead of silently truncating it
  • [SPARK-60297] Enforce TLS in SparkConnectClient when a token is set
  • [SPARK-60295] Fix flaky StreamingQueryTests.status test by waiting for query initialization before stop
  • [SPARK-60289] Upgrade gRPC Swift NIO Transport to 2.11.0
  • [SPARK-60290] Upgrade gRPC Swift Protobuf to 2.5.0
  • [SPARK-60292] Upgrade Swift Docker image to 6.4.1
  • [SPARK-60288] Upgrade grpc-swift-2 to 2.4.4
  • [SPARK-60064] Fix DataFrameReader and DataStreamReader to not send empty format when format is not specified
  • [SPARK-60063] Fix Catalog.listDatabases/getDatabase/getFunction to not crash on null locationUri and className
  • [SPARK-60061] Fix Row.getAsTimestampNanos and Row.== to not crash on out-of-range values
  • [SPARK-60060] Make StreamingQueryStatus and StreamingQueryException properties public
  • [SPARK-60055] Fix dropDuplicatesWithinWatermark to set withinWatermark in Deduplicate relation
  • [SPARK-60054] Fix DataFrame.hint to not crash on out-of-range integers and unsupported parameter types
  • [SPARK-60053] Fix MergeIntoWriter.update to pass assignments to the merge action
  • [SPARK-60051] Fix DataFrame.collect() to return Data for BINARY columns instead of crashing
  • [SPARK-59956] Use 4.0.5 RC1 for Spark 4.0 integration tests
  • [SPARK-59940] Use Swift 6.4 as the minimum version and unify CI on xcode-27
  • [SPARK-59904] Upgrade Iceberg to 1.12.0 in integration tests
  • [SPARK-59873] Use ubuntu-26.04 instead of ubuntu-latest in GitHub Actions
  • [SPARK-59867] Update GitHub Actions to the latest versions
  • [SPARK-59864] Use 4.1.4 RC1 for Spark 4.1 integration tests
  • [SPARK-59863] Use 4.2.1 RC1 for Spark 4.2 integration tests
  • [SPARK-59735] Use static-debian13 distroless image in Examples
  • [SPARK-59736] Use Swift 6.4 Docker image in all Linux CI jobs
  • [SPARK-59675] Migrate Examples to Swift 6.4
  • [SPARK-59664] Use official Swift 6.4 Docker image instead of nightly
  • [SPARK-59646] Upgrade gRPC Swift NIO Transport to 2.10.0
  • [SPARK-59592] Use Java 21 at least in CI
  • [SPARK-59591] Add integration tests with Swift 6.4
  • [SPARK-59590] Remove unnecessary try keywords reported by Swift 6.4
  • [SPARK-59583] Upgrade actions/checkout to v7
  • [SPARK-59485] Use server-side aggregation for DataFrame.count()
  • [SPARK-59484] Use value as the output column name of DataFrame.toJSON
  • [SPARK-59483] Fix DataFrame.(head|first) to throw an error instead of crashing on empty results
  • [SPARK-59481] Fix concurrent commands on the same session to return correct responses
  • [SPARK-59480] Fix concurrent actions on the same DataFrame to return correct results
  • [SPARK-59474] Support DECIMAL type in createDataFrame with [[Sendable?]] rows
  • [SPARK-59473] Support top-level Decimal decoding in DataFrame.collect(as:)
  • [SPARK-59472] Support Decimal properties in createDataFrame with Encodable types
  • [SPARK-59471] Fix DECIMAL decoding to use the full signed 128-bit value
  • [SPARK-59470] Respect spark.sql.caseSensitive when resolving column names in DataFrame.collect(as:)
  • [SPARK-59465] Support integral to floating-point upcasts in DataFrame.collect(as:)
  • [SPARK-59464] Fix DATE handling for dates before 1970 and after 2106
  • [SPARK-59463] Fix FlatBuffers nesting assertion on TIMESTAMP columns in createDataFrame
  • [SPARK-59462] Support safe upcasts and throw errors instead of crashing in DataFrame.collect(as:)
  • [SPARK-59450] Support Codable in createDataFrame and DataFrame.collect
  • [SPARK-59449] Support type-safe getters in Row
  • [SPARK-59429] Pass plan ID in DataFrame.colRegex
  • [SPARK-59420] Support withField and dropFields in Column
  • [SPARK-59416] Support col and subscript column accessors in DataFrame
  • [SPARK-59414] Assign plan IDs to DataFrame relations
  • [SPARK-59399] Support colRegex in DataFrame
  • [SPARK-59391] Support groupingSets in DataFrame
  • [SPARK-59380] Upgrade actions/setup-java to v6
  • [SPARK-59264] Cache Apache Spark distributions in GitHub Actions jobs
  • [SPARK-59260] Index all public functions and types in the DocC catalog
  • [SPARK-59258] Support higher-order map functions
  • [SPARK-59257] Update docker GitHub Actions to the latest ASF-approved hashes
  • [SPARK-59254] Support higher-order array functions
  • [SPARK-59245] Support HLL sketch and bitmap aggregate functions
  • [SPARK-59241] Support Theta, KLL and Tuple sketch functions
  • [SPARK-59240] Support call_function and call_udf functions
  • [SPARK-59239] Support counter_diff window function
  • [SPARK-59238] Support partition transform functions
  • [SPARK-59237] Support broadcast function
  • [SPARK-59236] Support vector functions
  • [SPARK-59235] Support error, reflection and bitmap functions
  • [SPARK-59228] Support AES and HMAC functions
  • [SPARK-59227] Support base32 functions
  • [SPARK-59226] Support geospatial ST functions
  • [SPARK-59225] Support URL functions
  • [SPARK-59223] Support VARIANT functions
  • [SPARK-59222] Support XML functions
  • [SPARK-59221] Support linear regression aggregate functions
  • [SPARK-59217] Support interval and timestamp constructor functions
  • [SPARK-59216] Support JSON and CSV functions
  • [SPARK-59215] Support general aggregate functions
  • [SPARK-59214] Support bitwise functions
  • [SPARK-59213] Support null ordering in sort functions and Column
  • [SPARK-59211] Support conditional functions
  • [SPARK-59210] Support misc metadata functions
  • [SPARK-59209] Support greatest, least, random and try_* arithmetic functions
  • [SPARK-59207] Support predicate functions
  • [SPARK-59206] Support hash functions
  • [SPARK-59205] Support TIMESTAMP_NTZ(p) and TIMESTAMP_LTZ(p) in createDataFrame
  • [SPARK-59203] Support nanosecond timestamp literals in lit and SQL parameter binding
  • [SPARK-59198] Support nanosecond timestamp values in DataFrame.collect
  • [SPARK-59197] Support TIMESTAMP_NTZ(p) and TIMESTAMP_LTZ(p) nanosecond timestamp types in DataType
  • [SPARK-59195] Create a gRPC channel per session instead of per RPC
  • [SPARK-59194] Use numeric Spark version comparison in tests
  • [SPARK-59193] Add firstOrThrow to avoid trapping on empty server responses
  • [SPARK-59192] Avoid force casts in DataFrame.collect
  • [SPARK-59191] Support grpc_max_message_size in connection string
  • [SPARK-59190] Make Row.getAsBool throw instead of trapping on type mismatch
  • [SPARK-59179] Add Xcode 27 (Swift 6.4) GitHub Action job
  • [SPARK-59178] Use Xcode 26.6 in CIs
  • [SPARK-59177] Use lower-bound version checks in SQLTests and SparkConnectClientTests
  • [SPARK-59159] Remove queriesForSpark4Only from SQLTests
  • [SPARK-59157] Upgrade grpc-swift-2 to 2.4.3
  • [SPARK-59156] Add MacOS integration test with Apache Spark 4.3.0 RC1
  • [SPARK-59155] Support TIME(p) precision in createDataFrame
  • [SPARK-58957] Upgrade gRPC Swift NIO Transport to 2.9.2
  • [SPARK-58956] Upgrade swift-system to 1.8.1
  • [SPARK-58667] Upgrade swift-system to 1.8.0
  • [SPARK-52747] Test timezone variations
  • [SPARK-52746] Test isLocal behavior explicitly for Spark 4.0 and 4.1+ servers
  • [SPARK-58652] Update Spark Connect-generated Swift source code with branch-4.3 (2026-08-07)
  • [SPARK-58653] Support typed StreamingQueryProgress in (last|recent)Progress
  • [SPARK-58651] Support parseDDL in SparkSession
  • [SPARK-58648] Support StructType in schema, createDataFrame, and to APIs
  • [SPARK-58647] Add public DataType and StructType
  • [SPARK-58644] Document intentional ignore of queryStartedEventJson in DataStreamWriter
  • [SPARK-58610] Support cloneSession in SparkSession
  • [SPARK-58608] Support Observation to retrieve observed metrics
  • [SPARK-58579] Support TIME type in SparkSession.createDataFrame
  • [SPARK-58574] Support TIME type literals in lit and SQL parameter binding
  • [SPARK-58571] Support TIME type values in DataFrame.collect
  • [SPARK-58568] Support TIME type in DataType.simpleString
  • [SPARK-58563] Support server-side error chain in SparkConnectError via FetchErrorDetails
  • [SPARK-58560] Use Apache Spark 4.0.4 and 4.1.3 in CI
  • [SPARK-58543] Support reattachable execution with ReattachExecute and ReleaseExecute
  • [SPARK-58542] Upgrade gRPC Swift NIO Transport to 2.9.1
  • [SPARK-58541] Support retry with exponential backoff in SparkConnectClient
  • [SPARK-58539] Support error details in SparkConnectError using gRPC ErrorInfo
  • [SPARK-58530] Support large local data in createDataFrame via CachedLocalRelation
  • [SPARK-58528] Support cache artifact upload in SparkConnectClient
  • [SPARK-58526] Fix VariableBufferBuilder to handle large first value and offsets truncation
  • [SPARK-58524] Add SHA256 struct
  • [SPARK-58498] Support geometry/geography types in DataType.simpleString
  • [SPARK-58497] Support createDataFrame in SparkSession
  • [SPARK-58496] Support field-name-based access in Row
  • [SPARK-58493] Support aggregate functions in AggregateFunctions
  • [SPARK-58492] Support window functions with Window, WindowSpec and Column.over
  • [SPARK-58491] Support expr in Functions
  • [SPARK-58490] Support collection functions in CollectionFunctions
  • [SPARK-58489] Support datetime functions in DateTimeFunctions
  • [SPARK-58483] Support string functions in StringFunctions
  • [SPARK-58479] Support math functions in MathFunctions
  • [SPARK-58477] Support when/otherwise in Column and Functions
  • [SPARK-58476] Support Column-based sort/withColumn/drop/join in DataFrame
  • [SPARK-58475] Support predicate and string methods in Column
  • [SPARK-58474] Support Swift literal operands in Column operators via SparkLiteral
  • [SPARK-58473] Support comparison/logical/arithmetic operators in Column
  • [SPARK-58434] Support pivot in GroupedData
  • [SPARK-58433] Support repartitionByRange for DataFrame
  • [SPARK-58432] Support sortWithinPartitions for DataFrame
  • [SPARK-58430] Support alias for DataFrame
  • [SPARK-58427] Support randomSplit for DataFrame
  • [SPARK-58425] Support observe for DataFrame
  • [SPARK-58424] Add count/sum/avg/mean/min/max to GroupedData
  • [SPARK-58421] Improve sql parameter binding to support more literal types instead of crashing
  • [SPARK-58345] Handle Arrow message load failures in DataFrame.execute
  • [SPARK-58344] Support Arrow Null type to fix collect() crash on void columns
  • [SPARK-58339] Send RemoveCachedRemoteRelationCommand when checkpointed DataFrame is deallocated
  • [SPARK-58338] Fix defineFlow to send relation via relation_flow_details
  • [SPARK-58337] Fix StreamingQuery.exception() to return nil when no exception exists
  • [SPARK-58336] Support Column expression type and functions API
Assets 2
Loading