Skip to content

[AMORO-6789]Memory leaks when using mix -hive format to write parquet files.#2820

Merged
zhoujinsong merged 1 commit intoapache:masterfrom
nicochen:master
May 9, 2024
Merged

[AMORO-6789]Memory leaks when using mix -hive format to write parquet files.#2820
zhoujinsong merged 1 commit intoapache:masterfrom
nicochen:master

Conversation

@nicochen
Copy link
Copy Markdown
Contributor

@nicochen nicochen commented May 8, 2024

Fix bug that memory leaks when using mix -hive format to write parquet files.

Why are the changes needed?

To avoid writing tasks causing memory leak when write a mix-formatted table. Details refers to issue pls.
Close #2789 .

Brief change log

  • add logic of release a compressor to pool after finishing writing a file.

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (no)
  • If yes, how is the feature documented? (not applicable)

@github-actions github-actions bot added the module:mixed-hive Hive moduel for Mixed Format label May 8, 2024
@nicochen
Copy link
Copy Markdown
Contributor Author

nicochen commented May 8, 2024

@zhoujinsong TBR

Copy link
Copy Markdown
Contributor

@majin1102 majin1102 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Copy Markdown
Contributor

@zhoujinsong zhoujinsong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Thanks for the contribution!

@zhoujinsong zhoujinsong merged commit 30b22b8 into apache:master May 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:mixed-hive Hive moduel for Mixed Format

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Memory leaks when using mix -hive format to write parquet files

3 participants