Skip to content

Conversation

@ASiegeLion
Copy link
Contributor

@ASiegeLion ASiegeLion commented Aug 29, 2023

Why are the changes needed?

Close #5210.

The cancel operation may cause log file handle leak

java    6664 hadoop 5280w      REG                8,6    456619     525555 /opt/apache-kyuubi-1.7.1-bin/work/server_operation_logs/c8903f6d-4714-4053-b510-d8aaee164405/84b48809-a9fa-4d42-89de-0bc8be66ebbc (deleted)
java    6664 hadoop 5281r      REG                8,6      3425     525587 /opt/apache-kyuubi-1.7.1-bin/work/server_operation_logs/33ba2700-4424-4fe4-a359-ac15b7657ff1/85f9ef2d-f136-4786-8fb0-6ef24c0f8b0f (deleted)
java    6664 hadoop 5282r      REG                8,6    456619     525555 /opt/apache-kyuubi-1.7.1-bin/work/server_operation_logs/c8903f6d-4714-4053-b510-d8aaee164405/84b48809-a9fa-4d42-89de-0bc8be66ebbc (deleted)
java    6664 hadoop 5283w      REG                8,6     29978     525531 /opt/apache-kyuubi-1.7.1-bin/work/server_operation_logs/3a34f9e6-c128-4c64-9714-0e43f77a56aa/1046d6b3-92d9-4554-9e91-2d5f634b2adf (deleted)
java    6664 hadoop 5284r      REG                8,6      3373     525549 /opt/apache-kyuubi-1.7.1-bin/work/server_operation_logs/31955d56-8b46-47a3-9740-4daef398341b/51709d10-3da2-4f5a-a54b-0dc6cd0bec24 (deleted)
java    6664 hadoop 5285r      REG                8,6     29978     525531 /opt/apache-kyuubi-1.7.1-bin/work/server_operation_logs/3a34f9e6-c128-4c64-9714-0e43f77a56aa/1046d6b3-92d9-4554-9e91-2d5f634b2adf (deleted)
java    6664 hadoop 5286w      REG                8,6    782279     525528 /opt/apache-kyuubi-1.7.1-bin/work/server_operation_logs/c7e8c954-be85-4eee-98cf-0e839ef1c972/c3ed1788-9cea-4503-b2bf-55450af5f9f7 (deleted)
java    6664 hadoop 5287r      REG                8,6    782279     525528 /opt/apache-kyuubi-1.7.1-bin/work/server_operation_logs/c7e8c954-be85-4eee-98cf-0e839ef1c972/c3ed1788-9cea-4503-b2bf-55450af5f9f7 (deleted)
java    6664 hadoop 5288r      REG                8,6      3372     525538 /opt/apache-kyuubi-1.7.1-bin/work/server_operation_logs/2e9d82f1-2d3e-4d7f-9e9f-423dcc3bb5ab/7b97e0ca-2ac9-464d-a6c5-cf8dd6fd5e3d (deleted)
java    6664 hadoop 5289w      REG                8,6      3372     525539 /opt/apache-kyuubi-1.7.1-bin/work/server_operation_logs/fdce08d0-447e-4ab0-8a80-81aac266ba17/92274ada-4d9d-4c50-b7d9-879ea54aee40 (deleted)
java    6664 hadoop 5290r      REG                8,6      3388     525552 /opt/apache-kyuubi-1.7.1-bin/work/server_operation_logs/0e30073a-1792-49a6-be30-d2cd8d2c3d46/c1d2d080-8a5a-4170-91b2-617d66f2ea24 (deleted)
java    6664 hadoop 5291r      REG                8,6      3372     525592 /opt/apache-kyuubi-1.7.1-bin/work/server_operation_logs/d891ec77-e5dc-4f33-8075-ed293c73bd4e/503ea45e-7740-4d6d-9ed0-5f243da70e38 (deleted)
java    6664 hadoop 5292w      REG                8,6      3364     525550 /opt/apache-kyuubi-1.7.1-bin/work/server_operation_logs/2071e068-d533-46b7-a76e-22f3d56db6ad/03c83e12-84b4-47b2-9413-3e7bd85913ea (deleted)
java    6664 hadoop 5293w      REG                8,6      2923     525529 /opt/apache-kyuubi-1.7.1-bin/work/server_operation_logs/8cbb20ad-8046-4ab6-90a8-51986944d5a0/d8aaa439-d5cd-4e86-bc75-dac1a9968502 (deleted)
java    6664 hadoop 5294r      REG                8,6      3364     525550 /opt/apache-kyuubi-1.7.1-bin/work/server_operation_logs/2071e068-d533-46b7-a76e-22f3d56db6ad/03c83e12-84b4-47b2-9413-3e7bd85913ea (deleted)
java    6664 hadoop 5295w      REG                8,6      3378     525561 /opt/apache-kyuubi-1.7.1-bin/work/server_operation_logs/9acf2cd5-aefc-4171-8bb0-cc01a1fb4429/db5458ba-d241-4415-86b5-116cbbdac75d (deleted)
java    6664 hadoop 5296w      REG                8,6      3368     525567 /opt/apache-kyuubi-1.7.1-bin/work/server_operation_logs/135715cc-69f1-44bb-9170-c8f128ec6f1a/6b382b1b-c02f-474b-9d62-f8ec49be8288 (deleted)
java    6664 hadoop 5297r      REG                8,6      3368     525567 /opt/apache-kyuubi-1.7.1-bin/work/server_operation_logs/135715cc-69f1-44bb-9170-c8f128ec6f1a/6b382b1b-c02f-474b-9d62-f8ec49be8288 (deleted)
java    6664 hadoop 5298w      REG                8,6      3372     525538 /opt/apache-kyuubi-1.7.1-bin/work/server_operation_logs/2e9d82f1-2d3e-4d7f-9e9f-423dcc3bb5ab/7b97e0ca-2ac9-464d-a6c5-cf8dd6fd5e3d (deleted)
java    6664 hadoop 5299w      REG                8,6      3378     525564 /opt/apache-kyuubi-1.7.1-bin/work/server_operation_logs/0a8cb7d2-6c9c-4bfa-9fe7-1a902e909cf3/97a643c8-b400-49b9-b48d-e4f2ce1b9099 (deleted)
java    6664 hadoop 5300w      REG                8,6    134928     525608 /opt/apache-kyuubi-1.7.1-bin/work/server_operation_logs/e858abae-403d-4cf7-b266-708551118489/66e2b15f-0a3f-4983-ba6a-844a99b7c106 (deleted)
java    6664 hadoop 5301r      REG                8,6      3098     525570 /opt/apache-kyuubi-1.7.1-bin/work/server_operation_logs/f2a0d5df-f1d0-4bb0-a793-702b7cdbbf4a/1b417bf1-8039-413b-9107-178514288358 (deleted)

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before make a pull request

Was this patch authored or co-authored using generative AI tooling?

@codecov-commenter
Copy link

codecov-commenter commented Aug 29, 2023

Codecov Report

Merging #5211 (6366e96) into master (b21ae88) will not change coverage.
The diff coverage is 0.00%.

@@          Coverage Diff           @@
##           master   #5211   +/-   ##
======================================
  Coverage    0.00%   0.00%           
======================================
  Files         590     590           
  Lines       33357   33357           
  Branches     4407    4407           
======================================
  Misses      33357   33357           
Files Changed Coverage Δ
.../org/apache/kyuubi/operation/KyuubiOperation.scala 0.00% <0.00%> (ø)

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

try {
// For launch engine operation, we use OperationLog to pass engine submit log but
// at that time we do not have remoteOpHandle
getOperationLog.foreach(_.close())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what happens if this operation is closed ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Close the log stream gracefully, then delete the log file

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I replied to your message but forgot to submit it

ulysses-you pushed a commit that referenced this pull request Sep 14, 2023
### _Why are the changes needed?_

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

Closes #5211 from ASiegeLion/master.

Closes #5210

6366e96 [peiyue liu] Merge branch 'apache:master' into master
34dcc57 [liupeiyue] [KYUUBI #5210]Cancel operation will cause the log file node to leak

Lead-authored-by: peiyue liu <945076608@qq.com>
Co-authored-by: liupeiyue <liupeiyue@yy.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
(cherry picked from commit 721c0a4)
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
ulysses-you pushed a commit that referenced this pull request Sep 14, 2023
### _Why are the changes needed?_

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

Closes #5211 from ASiegeLion/master.

Closes #5210

6366e96 [peiyue liu] Merge branch 'apache:master' into master
34dcc57 [liupeiyue] [KYUUBI #5210]Cancel operation will cause the log file node to leak

Lead-authored-by: peiyue liu <945076608@qq.com>
Co-authored-by: liupeiyue <liupeiyue@yy.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
(cherry picked from commit 721c0a4)
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
@ulysses-you
Copy link
Contributor

thanks, merged to master/1.8/1.7

@ulysses-you ulysses-you added this to the v1.7.2 milestone Sep 14, 2023
@pan3793 pan3793 modified the milestones: v1.7.2, 1.7.3 Sep 18, 2023
@pan3793 pan3793 changed the title [KYUUBI #5210]Cancel operation will cause the log file node to leak [KYUUBI #5210] Cancel operation will cause the log file node to leak Dec 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Cancel operation will cause the log file node to leak

4 participants