8114830: (fs) Files.copy fails due to interference from something else changing the file system #15141

bplb · 2023-08-03T16:20:02Z

Throw a FileSystemException if attempting to create the target file with O_EXCL fails with EEXIST.

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue

Issue

JDK-8114830: (fs) Files.copy fails due to interference from something else changing the file system (Bug - P4)

Reviewers

Alan Bateman (@AlanBateman - Reviewer)
Vyom Tewari (@vyommani - Committer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/15141/head:pull/15141
$ git checkout pull/15141

Update a local copy of the PR:
$ git checkout pull/15141
$ git pull https://git.openjdk.org/jdk.git pull/15141/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 15141

View PR using the GUI difftool:
$ git pr show -t 15141

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/15141.diff

Webrev

Link to Webrev Comment

…e changing the file system

bridgekeeper · 2023-08-03T16:21:40Z

👋 Welcome back bpb! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2023-08-03T16:23:56Z

@bplb The following label will be automatically applied to this pull request:

nio

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

mlbridge · 2023-08-03T16:27:50Z

Webrevs

bplb · 2023-08-03T16:55:12Z

The same problem occurs for directories. Investigating other cases and will update this PR as needed.

src/java.base/unix/classes/sun/nio/fs/UnixFileSystem.java

test/jdk/java/nio/file/Files/CopyInterference.java

…rvice

test/jdk/java/nio/file/Files/CopyInterference.java

src/java.base/unix/classes/sun/nio/fs/UnixFileSystem.java

test/jdk/java/nio/file/Files/CopyInterference.java

AlanBateman

The src change looks fine.

I've done a pass over the test, it looks okay but it would be good to get another set of eyes (if possible) and some confidence that it doesn't fail with repeated runs.

openjdk · 2023-08-10T02:11:31Z

@bplb This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8114830: (fs) Files.copy fails due to interference from something else changing the file system

Reviewed-by: alanb, vtewari

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 100 new commits pushed to the master branch:

ec0cc63: 8313904: [macos] All signing tests which verifies unsigned app images are failing
7332502: 8314139: TEST_BUG: runtime/os/THPsInThreadStackPreventionTest.java could fail on machine with large number of cores
8f1c134: 8313798: [aarch64] sun/tools/jhsdb/HeapDumpTestWithActiveProcess.java sometimes times out on aarch64
1232677: 8313244: NM flags handling in configure process
6ffc032: 8314113: G1: Remove unused G1CardSetInlinePtr::card_at
62adeb0: 8311648: Refactor the Arena/Chunk/ChunkPool interface
43462a3: 8313224: Avoid calling JavaThread::current() in MemAllocator::Allocation constructor
9abb2a5: 8312461: JNI warnings in SunMSCApi provider
42758cb: 8312882: Update the CONTRIBUTING.md with pointers to lifecycle of a PR
88b4e3b: 8304292: Memory leak related to ClassLoader::update_class_path_entry_list
... and 90 more: https://git.openjdk.org/jdk/compare/8248e351d0bed263fb68d8468004a4286e6391af...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

bplb · 2023-08-10T02:13:04Z

I've done a pass over the test, it looks okay but it would be good to get [...] some confidence that it doesn't fail with repeated runs.

I'll submit some runs with many repeats before integrating.

test/jdk/java/nio/file/Files/CopyInterference.java

vyommani

Latest changes looks OK to me, i tested fix at local env and it is working as expected.

bplb · 2023-08-11T17:36:06Z

I've done a pass over the test, it looks okay but it would be good to get [...] some confidence that it doesn't fail with repeated runs.

I'll submit some runs with many repeats before integrating.

300 repeats of the test on the usual four platforms in the CI passed.

bplb · 2023-08-14T17:46:23Z

/integrate

openjdk · 2023-08-14T17:48:54Z

Going to push as commit c132176.
Since your change was applied there have been 116 commits pushed to the master branch:

e56d3bc: 8313657: com.sun.jndi.ldap.Connection.cleanup does not close connections on SocketTimeoutErrors
4b2703a: 8313678: SymbolTable can leak Symbols during cleanup
f41c267: 8314045: ArithmeticException in GaloisCounterMode
911d1db: 8314078: HotSpotConstantPool.lookupField() asserts due to field changes in ConstantPool.cpp
6574dd7: 8314025: Remove JUnit-based test in java/lang/invoke from problem list
207bd00: 8313756: [BACKOUT] 8308682: Enhance AES performance
823f5b9: 8308850: Change JVM options with small ranges that get -Wconversion warnings to 32 bits
5bfb82e: 8314119: G1: Fix -Wconversion warnings in G1CardSetInlinePtr::card_pos_for
06aa3c5: 8314118: Update JMH devkit to 1.37
4164693: 8313372: [JVMCI] Export vmIntrinsics::is_intrinsic_available results to JVMCI compilers.
... and 106 more: https://git.openjdk.org/jdk/compare/8248e351d0bed263fb68d8468004a4286e6391af...master

Your commit was automatically rebased without conflicts.

openjdk · 2023-08-14T17:49:00Z

@bplb Pushed as commit c132176.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

turbanoff · 2023-08-22T17:54:26Z

Test CopyInterference fails for me on Windows 11

java.nio.file.FileSystemException: .\\foobargus39884821046272467\\link: A required privilege is not held by the client
	at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92)
	at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
	at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
	at java.base/sun.nio.fs.WindowsFileSystemProvider.createSymbolicLink(WindowsFileSystemProvider.java:604)
	at java.base/java.nio.file.Files.createSymbolicLink(Files.java:1069)
	at CopyInterference.pathAndOptionsProvider(CopyInterference.java:117)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
	at org.junit.jupiter.engine.execution.DefaultExecutableInvoker.invoke(DefaultExecutableInvoker.java:55)
	at org.junit.jupiter.params.provider.MethodArgumentsProvider.lambda$provideArguments$1(MethodArgumentsProvider.java:55)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:1024)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
	at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1709)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
	at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1709)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
	at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:110)
	at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:44)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
	at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
	at com.sun.javatest.regtest.agent.JUnitRunner.runWithJUnitPlatform(JUnitRunner.java:142)
	at com.sun.javatest.regtest.agent.JUnitRunner.main(JUnitRunner.java:95)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:333)
	at java.base/java.lang.Thread.run(Thread.java:1570)
	Suppressed: org.junit.platform.commons.PreconditionViolationException: Configuration error: You must configure at least one set of arguments for this @ParameterizedTest
		at org.junit.platform.commons.util.Preconditions.condition(Preconditions.java:299)
		at org.junit.jupiter.params.ParameterizedTestExtension.lambda$provideTestTemplateInvocationContexts$5(ParameterizedTestExtension.java:98)
		at java.base/java.util.stream.AbstractPipeline.close(AbstractPipeline.java:323)
		at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)
		... 55 more
JavaTest Message: JUnit Platform Failure(s): 1

bplb · 2023-08-22T17:55:54Z

A required privilege is not held

You need to run with elevated privileges.

AlanBateman · 2023-08-22T17:59:08Z

You need to run with elevated privileges.

This test should probably use TestUtil.supportsLinks to test if it can create a sym link, skip if it can't.

bplb · 2023-08-22T18:02:19Z

This test should probably use TestUtil.supportsLinks

You are correct. I'll file an issue to fix it.

bplb · 2023-08-22T18:11:41Z

https://bugs.openjdk.org/browse/JDK-8314810

bplb · 2023-08-22T18:27:12Z

#15391

8114830: (fs) Files.copy fails due to interference from something els…

f03dc3e

…e changing the file system

openjdk bot added the rfr Pull request is ready for review label Aug 3, 2023

openjdk bot added the nio nio-dev@openjdk.org label Aug 3, 2023

bplb marked this pull request as draft August 3, 2023 17:01

openjdk bot removed the rfr Pull request is ready for review label Aug 3, 2023

bplb added 2 commits August 3, 2023 14:13

8114830: Extend EEXIST check to directories, links, and devices

7011f73

8114830: Convert test to JUnit 5 and expand coverage

a69e8d6

bplb marked this pull request as ready for review August 3, 2023 21:22

openjdk bot added the rfr Pull request is ready for review label Aug 3, 2023

AlanBateman reviewed Aug 3, 2023

View reviewed changes

src/java.base/unix/classes/sun/nio/fs/UnixFileSystem.java Outdated Show resolved Hide resolved

8114830: Use single-arg FileSystemException ctor

171fbd2

AlanBateman reviewed Aug 4, 2023

View reviewed changes

src/java.base/unix/classes/sun/nio/fs/UnixFileSystem.java Show resolved Hide resolved

8114830: Add flags.replaceExisting to check for FileSystemException

bc43e5e

AlanBateman reviewed Aug 4, 2023

View reviewed changes

test/jdk/java/nio/file/Files/CopyInterference.java Outdated Show resolved Hide resolved

8114830: In test, use try-with-resources to closs to close ExecutorSe…

541f36b

…rvice

AlanBateman reviewed Aug 5, 2023

View reviewed changes

test/jdk/java/nio/file/Files/CopyInterference.java Outdated Show resolved Hide resolved

8114830: Minor CopyTask cleanup

f944036

vyommani reviewed Aug 9, 2023

View reviewed changes

src/java.base/unix/classes/sun/nio/fs/UnixFileSystem.java Outdated Show resolved Hide resolved

8114830: Correct toString() call

a1bbd21

AlanBateman reviewed Aug 10, 2023

View reviewed changes

test/jdk/java/nio/file/Files/CopyInterference.java Outdated Show resolved Hide resolved

AlanBateman reviewed Aug 10, 2023

View reviewed changes

test/jdk/java/nio/file/Files/CopyInterference.java Outdated Show resolved Hide resolved

8114830: Remove running.set(false) from catch bloblocks in test

028171a

AlanBateman approved these changes Aug 10, 2023

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Aug 10, 2023

vyommani reviewed Aug 10, 2023

View reviewed changes

test/jdk/java/nio/file/Files/CopyInterference.java Show resolved Hide resolved

8114830: Remove deleteOnExit() from test

fc604ec

vyommani approved these changes Aug 11, 2023

View reviewed changes

AlanBateman approved these changes Aug 13, 2023

View reviewed changes

openjdk bot added the integrated Pull request has been integrated label Aug 14, 2023

openjdk bot closed this Aug 14, 2023

openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Aug 14, 2023

bplb deleted the Files-copy-interference-8114830 branch August 14, 2023 17:52

8114830: (fs) Files.copy fails due to interference from something else changing the file system #15141

8114830: (fs) Files.copy fails due to interference from something else changing the file system #15141

Uh oh!

Conversation

bplb commented Aug 3, 2023 • edited by openjdk bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Progress

Issue

Reviewers

Reviewing

Webrev

Uh oh!

bridgekeeper bot commented Aug 3, 2023

Uh oh!

openjdk bot commented Aug 3, 2023

Uh oh!

mlbridge bot commented Aug 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Webrevs

Uh oh!

bplb commented Aug 3, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AlanBateman left a comment

Choose a reason for hiding this comment

Uh oh!

openjdk bot commented Aug 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bplb commented Aug 10, 2023

Uh oh!

Uh oh!

vyommani left a comment

Choose a reason for hiding this comment

Uh oh!

bplb commented Aug 11, 2023

Uh oh!

bplb commented Aug 14, 2023

Uh oh!

openjdk bot commented Aug 14, 2023

Uh oh!

openjdk bot commented Aug 14, 2023

Uh oh!

turbanoff commented Aug 22, 2023

Uh oh!

bplb commented Aug 22, 2023

Uh oh!

AlanBateman commented Aug 22, 2023

Uh oh!

bplb commented Aug 22, 2023

Uh oh!

bplb commented Aug 22, 2023

Uh oh!

bplb commented Aug 22, 2023

Uh oh!

Uh oh!

bplb commented Aug 3, 2023 •

edited by openjdk bot

Loading

mlbridge bot commented Aug 3, 2023 •

edited

Loading

openjdk bot commented Aug 10, 2023 •

edited

Loading