Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HADOOP-18764. fs.azure.buffer.dir to be under Yarn container path on yarn applications #5737

Merged
merged 1 commit into from
Jun 27, 2023
Merged

HADOOP-18764. fs.azure.buffer.dir to be under Yarn container path on yarn applications #5737

merged 1 commit into from
Jun 27, 2023

Conversation

mehakmeet
Copy link
Contributor

@mehakmeet mehakmeet commented Jun 12, 2023 •
edited
Loading

Description of PR

fs.azure.buffer.dir to be under Yarn container path on yarn applications

How was this patch tested?

mvn -Dparallel-tests=abfs -DtestsThreadCount=8 -Dscale clean verify on us-west-2 bucket

Results:

[INFO] Results:
[INFO] 
[WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 4

[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   ITestAzureBlobFileSystemLease.testTwoCreate:142  Expected to find 'There is currently a lease on the resource and no lease ID was specified in the request' but got unexpected exception: org.apache.hadoop.fs.PathIOException: `abfs://abfs-testcontainer-1b46713f-9625-4c06-8ecb-64b39c8dff3d@mmtusw.dfs.core.windows.net/fork-0002/test/testTwoCreate617053501b77/testfile': Input/output error: Parallel access to the create path detected. Failing request to honor single writer semantics
	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.checkException(AzureBlobFileSystem.java:1503)
	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.create(AzureBlobFileSystem.java:334)
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1231)
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1208)
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1089)
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1076)
	at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemLease.lambda$testTwoCreate$1(ITestAzureBlobFileSystemLease.java:144)
	at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:498)
	at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384)
	at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:453)
	at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemLease.testTwoCreate(ITestAzureBlobFileSystemLease.java:142)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)
Caused by: Parallel access to the create path detected. Failing request to honor single writer semantics
	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.conditionalCreateOverwriteFile(AzureBlobFileSystemStore.java:652)
	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.createFile(AzureBlobFileSystemStore.java:563)
	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.create(AzureBlobFileSystem.java:328)
	... 21 more

[ERROR]   ITestGetNameSpaceEnabled.testGetIsNamespaceEnabledWhenConfigIsFalse:98->unsetAndAssert:109 [getIsNamespaceEnabled should return the value configured for fs.azure.test.namespace.enabled] expected:<[fals]e> but was:<[tru]e>
[ERROR]   ITestGetNameSpaceEnabled.testGetIsNamespaceEnabledWhenConfigIsTrue:88->unsetAndAssert:109 [getIsNamespaceEnabled should return the value configured for fs.azure.test.namespace.enabled] expected:<[fals]e> but was:<[tru]e>
[ERROR]   ITestGetNameSpaceEnabled.testNonXNSAccount:77->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 Expecting getIsNamespaceEnabled() return false
[INFO] 
[ERROR] Tests run: 581, Failures: 4, Errors: 0, Skipped: 107

[INFO] Results:
[INFO] 
[WARNING] Tests run: 339, Failures: 0, Errors: 0, Skipped: 41

Seeing failures unrelated to the patch specifically ITestAzureBlobFileSystemLease.java since the other failures seem config related.

Is this a known failure?
CC @steveloughran

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@steveloughran
Copy link
Contributor

jira search shows that it is how the test fails for you; me i get timeouts more than anything else
#5134 (comment)

unrelated

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 51s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚 mvninstall 50m 3s trunk passed
+1 💚 compile 18m 40s trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚 compile 16m 57s trunk passed with JDK Private Build-1.8.0_362-8u372-gaus1-0ubuntu120.04-b09
+1 💚 mvnsite 1m 40s trunk passed
+1 💚 javadoc 1m 18s trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚 javadoc 0m 47s trunk passed with JDK Private Build-1.8.0_362-8u372-gaus1-0ubuntu120.04-b09
+1 💚 shadedclient 114m 2s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 59s the patch passed
+1 💚 compile 17m 41s the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚 javac 17m 41s the patch passed
+1 💚 compile 17m 8s the patch passed with JDK Private Build-1.8.0_362-8u372-gaus1-0ubuntu120.04-b09
+1 💚 javac 17m 8s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 1m 37s the patch passed
+1 💚 javadoc 1m 7s the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚 javadoc 0m 49s the patch passed with JDK Private Build-1.8.0_362-8u372-gaus1-0ubuntu120.04-b09
+1 💚 shadedclient 30m 5s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 18m 36s hadoop-common in the patch passed.
+1 💚 asflicense 1m 1s The patch does not generate ASF License warnings.
204m 23s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5737/1/artifact/out/Dockerfile
GITHUB PR #5737
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname Linux 4d9e5671baf5 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / e0f702c
Default Java Private Build-1.8.0_362-8u372-gaus1-0ubuntu120.04-b09
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-gaus1-0ubuntu120.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5737/1/testReport/
Max. process+thread count 1312 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5737/1/console
versions git=2.25.1 maven=3.6.3
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran
steveloughran approved these changes Jun 12, 2023
View reviewed changes
Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
+1

@mehakmeet mehakmeet merged commit 5db7107 into apache:trunk Jun 27, 2023
1 check failed
mehakmeet added a commit to mehakmeet/hadoop that referenced this pull request Jun 27, 2023
@mehakmeet
HADOOP-18764. fs.azure.buffer.dir to be under Yarn container path on …
73a7302 
…yarn applications (apache#5737)


Changing fs.azure.buffer.dir for azure so things clean up better in long-lived yarn clusters.

Contributed by: Mehakmeet Singh
@mehakmeet mehakmeet mentioned this pull request Jun 27, 2023
Merged
4 tasks
jiajunmao pushed a commit to jiajunmao/hadoop-MLEC that referenced this pull request Feb 6, 2024
@mehakmeet @jiajunmao
HADOOP-18764. fs.azure.buffer.dir to be under Yarn container path on …
7e7d160 
…yarn applications (apache#5737)


Changing fs.azure.buffer.dir for azure so things clean up better in long-lived yarn clusters.

Contributed by: Mehakmeet Singh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@steveloughran steveloughran steveloughran approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@mehakmeet @steveloughran @hadoop-yetus