Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Improvement] Support System.gc() with periodic GC interval #4352

Closed
2 of 4 tasks
turboFei opened this issue Feb 17, 2023 · 5 comments
Closed
2 of 4 tasks

[Improvement] Support System.gc() with periodic GC interval #4352

turboFei opened this issue Feb 17, 2023 · 5 comments

Comments

@turboFei
Copy link
Member

turboFei commented Feb 17, 2023 •
edited
Loading

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

What would you like to be improved?

Invoke System.gc() with periodic GC interval.

Refer spark ContextCleaner::periodicGCService

How should we improve?

Refer: https://spark.apache.org/docs/latest/configuration.html#memory-management

image

Are you willing to submit PR?

  • Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
  • No. I cannot submit a PR at this time.
@turboFei
Copy link
Member Author

how do you think about? cc @pan3793 @yaooqinn

also cc @lightning-L

@turboFei
Copy link
Member Author

FYI:
The memory usage is high, after I execute jmap -histo:live, the memory usage is low.

Note that: jmap -histo:live will trigger gc.

image

@turboFei
Copy link
Member Author

@pan3793 how do you think about?

@bowenliang123
Copy link
Contributor

+1

@pan3793
Copy link
Member

pan3793 commented Feb 18, 2023

+1 for introducing this feature

lightning-L added a commit to lightning-L/kyuubi that referenced this issue Feb 20, 2023
@lightning-L
[KYUUBI apache#4352] Support System.gc() with periodic GC interval
75cdfbb
turboFei pushed a commit that referenced this issue Feb 21, 2023
@lightning-L @turboFei
[KYUUBI #4352] Support System.gc() with periodic GC interval
2db326c 
### _Why are the changes needed?_

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4382 from lightning-L/kyuubi-4352.

Closes #4352

2beb1ef [Tianlin Liao] update config name
a9eb126 [Tianlin Liao] update config setting
75cdfbb [Tianlin Liao] [KYUUBI #4352] Support System.gc() with periodic GC interval

Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
(cherry picked from commit e5ade90)
Signed-off-by: fwang12 <fwang12@ebay.com>
SusurHe pushed a commit to SusurHe/incubator-kyuubi that referenced this issue Feb 21, 2023
Merge branch 'master' of github.com:apache/kyuubi
4cd3d23 
* 'master' of github.com:apache/kyuubi: (1557 commits)
  [KYUUBI apache#3951][FOLLOWUP] Audit the rest request params
  [KYUUBI apache#4377] Grant execute permission to release scripts
  [KYUUBI apache#4374] Release uploading should include kyuubi-spark-connector-hive
  [KYUUBI apache#4267] Show warning if SessionHandle is invalid
  [KYUUBI apache#4385] [DOCS] Refine release process
  [KYUUBI apache#4352] Support System.gc() with periodic GC interval
  [KYUUBI apache#4152][FOLLOWUP] LDAP configurations should be server-only
  [KYUUBI apache#4373] Using SVN_STAGING_REPO instead of SVN_STAGING_REPO in the release script to fix echo message
  [KYUUBI apache#4372] Support to return null value for OperationsResource rowset
  [KYUUBI apache#4371] Fix typo in `kyuubi_ecosystem.drawio`
  [KYUUBI apache#4216] Support to transfer client version for kyuubi hive jdbc and rest client sdk
  [KYUUBI apache#4345] Add the doc of kyuubi trino server
  [KYUUBI apache#3081][DOCS] Add Hudi connector doc in Trino
  [KYUUBI apache#4357] Bump Jersey from 2.38 to 2.39
  [KYUUBI apache#4338][FOLLOWUP] Fix K8s integration tests
  [KYUUBI apache#4326] [ARROW] Fix Spark session timezone format in arrow-based result format
  [KYUUBI apache#4360][FOLLOWUP] Get valid unlimited users from existing limiters instead of conf
  [KYUUBI apache#4362] Add `_configurations` in kerberos.rst
  [KYUUBI apache#4338] Bump Spark from 3.3.1 to 3.3.2
  [KYUUBI apache#4119][FOLLOWUP] Add app start time for batch api docs
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@bowenliang123 @turboFei @pan3793