Skip to content

[SPARK-56690][SQL] - Expose common TaskMemoryManager API on HashedRelation to avoid code duplication#55640

Open
erenavsarogullari wants to merge 1 commit intoapache:masterfrom
erenavsarogullari:SPARK-56690
Open

[SPARK-56690][SQL] - Expose common TaskMemoryManager API on HashedRelation to avoid code duplication#55640
erenavsarogullari wants to merge 1 commit intoapache:masterfrom
erenavsarogullari:SPARK-56690

Conversation

@erenavsarogullari
Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Currently, HashedRelation creates TaskMemoryManager by duplicating same TaskMemoryManager definition on following APIs:

HashedRelation.apply()
UnsafeHashedRelation.read()
LongToUnsafeRowMap.this()

It will be useful to expose a single TaskMemoryManager creation API to avoid code duplication and this approach will be useful for the following kind of cases by managing the required logic with single definition:
https://issues.apache.org/jira/browse/SPARK-54354
#53065

Why are the changes needed?

It will be useful for long-term maintenance.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Being added new UT case and existing test cases creating HashedRelation instance.

Was this patch authored or co-authored using generative AI tooling?

No

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant