Skip to content

[SPARK-46563][SQL] View simpleString not follow conf spark.sql.debug.maxToStringFields#44558

Closed
AngersZhuuuu wants to merge 3 commits intoapache:masterfrom
AngersZhuuuu:SPARK-46563
Closed

[SPARK-46563][SQL] View simpleString not follow conf spark.sql.debug.maxToStringFields#44558
AngersZhuuuu wants to merge 3 commits intoapache:masterfrom
AngersZhuuuu:SPARK-46563

Conversation

@AngersZhuuuu
Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

View simpleString not follow conf spark.sql.debug.maxToStringFields

Why are the changes needed?

When columns is too much, the info is noisy. View simpleString not follow conf spark.sql.debug.maxToStringFields

Does this PR introduce any user-facing change?

Yea, view's plan will follow spark.sql.debug.maxToStringFields

How was this patch tested?

No

Was this patch authored or co-authored using generative AI tooling?

No

@github-actions github-actions bot added the SQL label Jan 2, 2024
Copy link
Copy Markdown
Member

@MaxGekk MaxGekk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AngersZhuuuu Could you check other places, please. I have found a couple more places where we could restrict the number of outputted fields:

case class ExternalRDDScanExec[T](
...
  override def simpleString(maxFields: Int): String = {
    s"$nodeName${output.mkString("[", ",", "]")}"
  }

and

case class UnresolvedTableSpec(
...
  override def simpleString(maxFields: Int): String = {
    this.copy(properties = Utils.redact(properties).toMap).toString
  }

Copy link
Copy Markdown
Member

@MaxGekk MaxGekk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Waiting for CI.

@AngersZhuuuu
Copy link
Copy Markdown
Contributor Author

@AngersZhuuuu Could you check other places, please. I have found a couple more places where we could restrict the number of outputted fields:

case class ExternalRDDScanExec[T](
...
  override def simpleString(maxFields: Int): String = {
    s"$nodeName${output.mkString("[", ",", "]")}"
  }

and

case class UnresolvedTableSpec(
...
  override def simpleString(maxFields: Int): String = {
    this.copy(properties = Utils.redact(properties).toMap).toString
  }

Yea, let me check

@AngersZhuuuu
Copy link
Copy Markdown
Contributor Author

Checked all and done

@AngersZhuuuu
Copy link
Copy Markdown
Contributor Author

The remain failed GA seems not related?

@MaxGekk
Copy link
Copy Markdown
Member

MaxGekk commented Jan 3, 2024

The remain failed GA seems not related?

Seems not.

+1, LGTM. Merging to master.
Thank you, @AngersZhuuuu and @yaooqinn @HyukjinKwon for review.

@MaxGekk MaxGekk closed this in f844e78 Jan 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants