Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AUDIT][SPARK-42779][SQL] Allow V2 writes to indicate advisory shuffle partition size #8680

Closed
nartal1 opened this issue Jul 7, 2023 · 1 comment
Labels
audit_3.5.0 feature request New feature or request

Comments

@nartal1
Copy link
Collaborator

nartal1 commented Jul 7, 2023

Below PR adds an API for data sources to indicate the advisory partition size for V2 writes. Even though this is applicable only when AQE is enabled and we don't have code where we support advisorySize, we may have to update the case class of GpuShuffleExchageExec for Spark-3.5+ since it's updated now

case class ShuffleExchangeExec(
    override val outputPartitioning: Partitioning,
    child: SparkPlan,
    shuffleOrigin: ShuffleOrigin = ENSURE_REQUIREMENTS,
    advisoryPartitionSize: Option[Long] = None)

Apache Spark commit - apache/spark@59e0066dc7
Follow-on issue - apache/spark@2fece076a3

@nartal1 nartal1 added feature request New feature or request ? - Needs Triage Need team to review and classify audit_3.5.0 labels Jul 7, 2023
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Jul 11, 2023
@razajafri
Copy link
Collaborator

Fixed as part of #8969

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
audit_3.5.0 feature request New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants