[SPARK-56700][SS] Make DataStreamReader.name public#55651
Closed
ericm-db wants to merge 4 commits into
Closed
Conversation
Member
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
Could you make the CI happy, @ericm-db ?
Scalastyle checks passed.
The scalafmt check failed on sql/connect or sql/connect at following occurrences:
org.apache.maven.plugin.MojoExecutionException: Scalafmt: Unformatted files found
Error: Failed to execute goal org.antipathy:mvn-scalafmt_2.13:1.1.1713302731.c3d0074:format (default-cli) on project spark-sql-api_2.13: Error formatting Scala files: Scalafmt: Unformatted files found -> [Help 1]
Before submitting your change, please make sure to format your code using the following command:
./build/mvn scalafmt:format -Dscalafmt.skip=false -Dscalafmt.validateOnly=false -Dscalafmt.changedOnly=false -pl sql/api -pl sql/connect/common -pl sql/connect/server -pl sql/connect/shims -pl sql/connect/client/jvm
HyukjinKwon
approved these changes
May 4, 2026
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ericm-db
force-pushed
the
datastreamreader-name-public
branch
from
fa3df24 to
40bef88
Compare
anishshri-db
pushed a commit
that referenced
this pull request
May 6, 2026
### What changes were proposed in this pull request? Remove the `private[sql]` access modifier from `DataStreamReader.name` and add the method as a public abstract API to the `DataStreamReader` base class. - Added abstract `def name(sourceName: String): this.type` to the API base class (`sql/api/.../DataStreamReader.scala`) - Changed both classic and connect implementations from `private[sql] def name` to `override def name` - Moved Scaladoc to the base class; implementations use `inheritdoc` ### Why are the changes needed? The `name` method was introduced in SPARK-56453 as `private[sql]` while the API was being finalized. Now that the feature is ready, making it public allows users to assign names to streaming sources for stable checkpoint metadata and source evolution. ### Does this PR introduce _any_ user-facing change? Yes. `DataStreamReader.name(sourceName)` is now a public `Experimental` API available to all users. Previously it was package-private to `org.apache.spark.sql`. ### How was this patch tested? Existing tests cover the `name` functionality. This change only modifies the access level. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 4.6) Closes #55651 from ericm-db/datastreamreader-name-public. Authored-by: ericm-db <eric.marnadi@databricks.com> Signed-off-by: Anish Shrigondekar <anish.shrigondekar@databricks.com> (cherry picked from commit 0af3d42) Signed-off-by: Anish Shrigondekar <anish.shrigondekar@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Remove the
private[sql]access modifier fromDataStreamReader.nameand add the method as a public abstract API to theDataStreamReaderbase class.def name(sourceName: String): this.typeto the API base class (sql/api/.../DataStreamReader.scala)private[sql] def nametooverride def name@inheritdocWhy are the changes needed?
The
namemethod was introduced in SPARK-56453 asprivate[sql]while the API was being finalized. Now that the feature is ready, making it public allows users to assign names to streaming sources for stable checkpoint metadata and source evolution.Does this PR introduce any user-facing change?
Yes.
DataStreamReader.name(sourceName)is now a public@ExperimentalAPI available to all users. Previously it was package-private toorg.apache.spark.sql.How was this patch tested?
Existing tests cover the
namefunctionality. This change only modifies the access level.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.6)