-
Notifications
You must be signed in to change notification settings - Fork 8.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HADOOP-19052.Hadoop use Shell command to get the count of the hard link which takes a lot of time #6527
Conversation
…nk which takes a lot of time
🎊 +1 overall
This message was automatically generated. |
@@ -219,6 +230,16 @@ public void testGetLinkCount() throws IOException { | |||
assertEquals(1, getLinkCount(x3)); | |||
} | |||
|
|||
@Test | |||
public void testGetLinkCountFromFileAttribute() throws IOException { | |||
assertTrue(supportsHardLink(x1)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
supportsHardLink(x1) may be false if os is not unix style
@@ -220,6 +226,10 @@ public static int getLinkCount(File fileName) throws IOException { | |||
throw new FileNotFoundException(fileName + " not found."); | |||
} | |||
|
|||
if (Files.getFileStore(fileName.toPath()).supportsFileAttributeView(FileAttributeView)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Files.getFileStore(fileName.toPath()).supportsFileAttributeView(FileAttributeView) should be abstracted into a method, and be tested it in UT.
…nk which takes a lot of time
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
Description of PR
As described in HADOOP_19052. When we try to append a file, we will execute method
getHardLinkCount
twice. When we execute methodgetHardLinkCount
java will start a new process to execute a shell command and wait for it to fork. When the QPS ofappend
execution is very high, methodgetHardLinkCount
will take a long time to finish which will cause a long-time wait to acquire lock.I used another method to get the linkCount of a file whose file store supports the file attributes identified by the given file attribute view. This method does not start a new process and will finish in very short time even if the QPS of
append
execution is high.How was this patch tested?
I add a new UT testGetLinkCountFromFileAttribute and a public method supportsHardLink to get whether or not this file store supports the file attributes identified by the given file attribute view.
For code changes:
LICENSE
,LICENSE-binary
,NOTICE-binary
files?