Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-23547][SQL]Cleanup the .pipeout file when the Hive Session closed #20702

Closed
wants to merge 2 commits into from
Closed

[SPARK-23547][SQL]Cleanup the .pipeout file when the Hive Session closed #20702

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d965ba2
[SPARK-23547][SQL]Cleanup the .pipeout file when the Hive Session closed
Mar 1, 2018
93e87f5
fix code review
Mar 5, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
21 changes: 21 additions & 0 deletions sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java
View file
Open in desktop
Expand Up @@ -641,6 +641,8 @@ public void close() throws HiveSQLException {
opHandleSet.clear();
// Cleanup session log directory.
cleanupSessionLogDir();
// Cleanup pipeout file.
cleanupPipeoutFile();
HiveHistory hiveHist = sessionState.getHiveHistory();
if (null != hiveHist) {
hiveHist.closeStream();
Expand All @@ -665,6 +667,25 @@ public void close() throws HiveSQLException {
}
}

private void cleanupPipeoutFile() {
String lScratchDir = hiveConf.getVar(ConfVars.LOCALSCRATCHDIR);
String sessionID = hiveConf.getVar(ConfVars.HIVESESSIONID);

File[] fileAry = new File(lScratchDir).listFiles(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: There is an overloaded version of listfiles.
Using it saves 1 line.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry i did not find the overloaded version of listfiles which saves 1 line.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No problem. I hope this works.

File[] fileAry = new File(lScratchDir).listFiles(
      (dir, name) -> name.startsWith(sessionID) && name.endsWith(".pipeout"));

I think it would be good if you would have one unit test for your change.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@attilapiros will fix it, thanks!

file -> {
String name = file.getName();
return (name.startsWith(sessionID) && name.endsWith(".pipeout"));
});

for (File file : fileAry) {
try {
FileUtils.forceDelete(file);
} catch (Exception e) {
LOG.error("Failed to cleanup pipeout file: " + file, e);
}
}
}

private void cleanupSessionLogDir() {
if (isOperationLogEnabled) {
try {
Expand Down