[SPARK-59143] Close remaining PySpark 4.2.0 Spark Connect public-API parity gaps - #95
Closed
HyukjinKwon wants to merge 2 commits into
Closed
Conversation
…parity gaps A close introspection diff of the drop-in against the reference pyspark==4.2.0 Spark Connect client surfaced a small set of remaining public-API gaps. Close them. Import-path gaps (thin re-export shims mirroring the existing connect/dataframe.py): - Add the missing pyspark.sql.connect.* submodules: catalog, conf, group, window, readwriter, tvf, udf, udtf, observation, merge, streaming.readwriter, streaming.query. Reference code and the official Connect test suite import from these paths. - Add the public pyspark.sql.conf (RuntimeConfig, aliasing the Connect RuntimeConf) and pyspark.sql.tvf modules. Signature gaps vs reference: - StructType.treeString gains the maxDepth parameter (default 2147483647), wired through a new DataType::tree_string_with_depth in the core; tree_string() is kept. - StructType/StructField.fromJson parameter renamed data -> json. - TableValuedFunction.json_tuple / stack now take varargs *fields. - VariantType.toInternal parameter renamed obj -> variant. Add offline tests for the new import paths and the signature fixes; the signature tests were validated against reference pyspark 4.2.0 behavior. Co-authored-by: Isaac <no-reply@databricks.com>
zhengruifeng
approved these changes
Sep 1, 2026
…ested field names
A struct-typed field was rendered with simple_string() in DataType::append_tree,
which inlines the nested fields (e.g. "struct<inner:int>"). That leaked the child
names into the parent's type text, so treeString(maxDepth) could not truncate them
(and at full depth printed them twice). Render a struct field as the bare type name
("struct") and let the depth-controlled recursion show its children, matching
PySpark's StructType._build_formatted_string. Non-struct fields keep simple_string().
Co-authored-by: Isaac <no-reply@databricks.com>
HyukjinKwon
added a commit
that referenced
this pull request
Sep 1, 2026
…parity gaps ## What changes were proposed in this pull request? Closes the remaining PySpark 4.2.0 Spark Connect public-API parity gaps found by a close `inspect.signature` diff of the drop-in against the reference `pyspark==4.2.0` Connect client. **Import-path gaps** — thin re-export shims, same pattern as the existing `connect/dataframe.py`: - New `pyspark.sql.connect.*` submodules: `catalog`, `conf`, `group`, `window`, `readwriter`, `tvf`, `udf`, `udtf`, `observation`, `merge`, `streaming.readwriter`, `streaming.query`. The reference client and the official Connect test suite import from these paths, which previously raised `ImportError`. - New public `pyspark.sql.conf` (exposing `RuntimeConfig`, aliasing the Connect `RuntimeConf`) and `pyspark.sql.tvf` modules. **Signature gaps** vs the reference Connect client: - `StructType.treeString` gains the `maxDepth` parameter (default `2147483647`), wired through a new `DataType::tree_string_with_depth` in the core; `tree_string()` is preserved for existing callers. - `StructType.fromJson` / `StructField.fromJson` parameter renamed `data` → `json`. - `TableValuedFunction.json_tuple` / `stack` now take varargs `*fields`. - `VariantType.toInternal` parameter renamed `obj` → `variant`. The `pyspark.sql.functions` surface was already at full parity (all 529 reference functions, no signature mismatches) and is unchanged. ## Why are the changes needed? The package advertises drop-in parity with the PySpark 4.2.0 Spark Connect client. These were the residual deviations where reference-valid imports and calls could break. ## Does this PR introduce any user-facing change? Yes, additive: previously-failing imports now resolve, and the four methods above accept the reference parameter names/forms. ## How was this patch tested? New offline tests in `python/tests/test_dropin_offline.py`: - `test_connect_submodule_import_paths` (14 paths) — passes against the built extension. - Behavior tests for `treeString(maxDepth)`, `fromJson(json=...)`, and `VariantType.toInternal(variant=...)` — validated against reference `pyspark==4.2.0` to confirm they encode the correct behavior; they exercise the rebuilt extension under CI. The Rust changes were authored but not compiled on the dev box (offline); the build + official Connect suite in CI is the build/behavior gate. This pull request and its description were written by Isaac. Closes #95 from HyukjinKwon/SPARK-59143-close-connect-parity-gaps. Authored-by: Hyukjin Kwon <hyukjin.kwon@databricks.com> Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com> (cherry picked from commit 254dd50) Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Closes the remaining PySpark 4.2.0 Spark Connect public-API parity gaps found by a
close
inspect.signaturediff of the drop-in against the referencepyspark==4.2.0Connect client.
Import-path gaps — thin re-export shims, same pattern as the existing
connect/dataframe.py:pyspark.sql.connect.*submodules:catalog,conf,group,window,readwriter,tvf,udf,udtf,observation,merge,streaming.readwriter,streaming.query. The reference client and the officialConnect test suite import from these paths, which previously raised
ImportError.pyspark.sql.conf(exposingRuntimeConfig, aliasing the ConnectRuntimeConf) andpyspark.sql.tvfmodules.Signature gaps vs the reference Connect client:
StructType.treeStringgains themaxDepthparameter (default2147483647),wired through a new
DataType::tree_string_with_depthin the core;tree_string()is preserved for existing callers.StructType.fromJson/StructField.fromJsonparameter renameddata→json.TableValuedFunction.json_tuple/stacknow take varargs*fields.VariantType.toInternalparameter renamedobj→variant.The
pyspark.sql.functionssurface was already at full parity (all 529 referencefunctions, no signature mismatches) and is unchanged.
Why are the changes needed?
The package advertises drop-in parity with the PySpark 4.2.0 Spark Connect client.
These were the residual deviations where reference-valid imports and calls could
break.
Does this PR introduce any user-facing change?
Yes, additive: previously-failing imports now resolve, and the four methods above
accept the reference parameter names/forms.
How was this patch tested?
New offline tests in
python/tests/test_dropin_offline.py:test_connect_submodule_import_paths(14 paths) — passes against the built extension.treeString(maxDepth),fromJson(json=...), andVariantType.toInternal(variant=...)— validated against referencepyspark==4.2.0to confirm they encode the correct behavior; they exercise therebuilt extension under CI.
The Rust changes were authored but not compiled on the dev box (offline); the
build + official Connect suite in CI is the build/behavior gate.
This pull request and its description were written by Isaac.