Bazel server: ensure file streams are closed #5512

laszlocsomor · 2018-07-04T11:46:48Z

Description of the problem / feature request:

The Bazel server (the Java code) sometimes opens streams without closing them, relying on the garbage collector to finalize the InputStream and close the file. This leads to Bazel holding on to files and (on Windows) being unable to delete them later.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Examples:

bazel/src/main/java/com/google/devtools/build/lib/vfs/FileSystem.java

Lines 270 to 276 in 52035c1

    
           protected byte[] getDigest(final Path path, DigestHashFunction hashFunction) throws IOException { 
        
             return new ByteSource() { 
        
               @Override 
        
               public InputStream openStream() throws IOException { 
        
                 return getInputStream(path); 
        
               } 
        
             }.hash(hashFunction.getHash()).asBytes();

bazel/src/main/java/com/google/devtools/build/lib/analysis/actions/LauncherFileWriteAction.java

Line 78 in 52035c1

InputStream in = ctx.getInputPath(this.launcher).getInputStream();

bazel/src/main/java/com/google/devtools/build/lib/bazel/rules/android/AndroidSdkRepositoryFunction.java

Line 314 in 52035c1

properties.load(sourcePropertiesFilePath.getInputStream());

bazel/src/main/java/com/google/devtools/build/lib/bazel/rules/android/SdkMavenRepository.java

Line 203 in 52035c1

    
           DocumentBuilderFactory.newInstance().newDocumentBuilder().parse(path.getInputStream());

Correct usage would be try-with-resources:

bazel/src/main/java/com/google/devtools/build/lib/exec/BinTools.java

Line 242 in 52035c1

try (InputStream in = path.getInputStream()) {

What's the output of `bazel info release`?

release 0.15.0

The text was updated successfully, but these errors were encountered:

Use try-with-resources to ensure InputStreams that we open via FileSystem.InputStream(path) are closed. Eagerly closing InputStreams avoids hanging on to file handles until the garbage collector finalizes the InputStream, meaning Bazel on Windows (and other processes) can delete or mutate these files. Hopefully this avoids intermittent file deletion errors that sometimes occur on Windows. See #5512 RELNOTES: none PiperOrigin-RevId: 203338148

Use try-with-resources to ensure OutputStreams that we open via FileSystem.OutputStream(path) are closed. Eagerly closing OutputStreams avoids hanging on to file handles until the garbage collector finalizes the OutputStream, meaning Bazel on Windows (and other processes) can delete or mutate these files. Hopefully this avoids intermittent file deletion errors that sometimes occur on Windows. See #5512 RELNOTES: none PiperOrigin-RevId: 203342889

Use the "with" statement to open files in various Python scripts used by Bazel, to ensure these files are closed eagerly. See #5512 RELNOTES: none PiperOrigin-RevId: 203346678

Follow-up to commit 59f17d6. Use try-with-resources to ensure Reader objects are closed eagerly. Eagerly closing Readers avoids hanging on to file handles until the garbage collector finalizes the object, meaning Bazel on Windows (and other processes) can delete or mutate these files. Hopefully this avoids intermittent file deletion errors that sometimes occur on Windows. See #5512 RELNOTES: none PiperOrigin-RevId: 203771262

Follow-up to commit 09d2031. Use try-with-resources to ensure Writer objects are closed eagerly. Eagerly closing Writers avoids hanging on to file handles until the garbage collector finalizes the object, meaning Bazel on Windows (and other processes) can delete or mutate these files. Hopefully this avoids intermittent file deletion errors that sometimes occur on Windows. See #5512 RELNOTES: none PiperOrigin-RevId: 203934471

laszlocsomor · 2018-07-18T09:48:09Z

I think this is all fixed now.

Use try-with-resources to ensure InputStreams that we open via FileSystem.InputStream(path) are closed. Eagerly closing InputStreams avoids hanging on to file handles until the garbage collector finalizes the InputStream, meaning Bazel on Windows (and other processes) can delete or mutate these files. Hopefully this avoids intermittent file deletion errors that sometimes occur on Windows. See bazelbuild#5512 RELNOTES: none PiperOrigin-RevId: 203338148

Use try-with-resources to ensure OutputStreams that we open via FileSystem.OutputStream(path) are closed. Eagerly closing OutputStreams avoids hanging on to file handles until the garbage collector finalizes the OutputStream, meaning Bazel on Windows (and other processes) can delete or mutate these files. Hopefully this avoids intermittent file deletion errors that sometimes occur on Windows. See bazelbuild#5512 RELNOTES: none PiperOrigin-RevId: 203342889

Use the "with" statement to open files in various Python scripts used by Bazel, to ensure these files are closed eagerly. See bazelbuild#5512 RELNOTES: none PiperOrigin-RevId: 203346678

Follow-up to commit 59f17d6. Use try-with-resources to ensure Reader objects are closed eagerly. Eagerly closing Readers avoids hanging on to file handles until the garbage collector finalizes the object, meaning Bazel on Windows (and other processes) can delete or mutate these files. Hopefully this avoids intermittent file deletion errors that sometimes occur on Windows. See bazelbuild#5512 RELNOTES: none PiperOrigin-RevId: 203771262

Follow-up to commit 09d2031. Use try-with-resources to ensure Writer objects are closed eagerly. Eagerly closing Writers avoids hanging on to file handles until the garbage collector finalizes the object, meaning Bazel on Windows (and other processes) can delete or mutate these files. Hopefully this avoids intermittent file deletion errors that sometimes occur on Windows. See bazelbuild#5512 RELNOTES: none PiperOrigin-RevId: 203934471

Follow-up to commit 09d20311d982606093ed881d779bb05a5ee70ed3. Use try-with-resources to ensure Writer objects are closed eagerly. Eagerly closing Writers avoids hanging on to file handles until the garbage collector finalizes the object, meaning Bazel on Windows (and other processes) can delete or mutate these files. Hopefully this avoids intermittent file deletion errors that sometimes occur on Windows. See bazelbuild/bazel#5512 RELNOTES: none PiperOrigin-RevId: 203934471

Follow-up to commit 59f17d6e0550bf63a0b6ef182e2d63474e058ede. Use try-with-resources to ensure Reader objects are closed eagerly. Eagerly closing Readers avoids hanging on to file handles until the garbage collector finalizes the object, meaning Bazel on Windows (and other processes) can delete or mutate these files. Hopefully this avoids intermittent file deletion errors that sometimes occur on Windows. See bazelbuild/bazel#5512 RELNOTES: none PiperOrigin-RevId: 203771262

Use try-with-resources to ensure OutputStreams that we open via FileSystem.OutputStream(path) are closed. Eagerly closing OutputStreams avoids hanging on to file handles until the garbage collector finalizes the OutputStream, meaning Bazel on Windows (and other processes) can delete or mutate these files. Hopefully this avoids intermittent file deletion errors that sometimes occur on Windows. See bazelbuild/bazel#5512 RELNOTES: none PiperOrigin-RevId: 203342889

Use try-with-resources to ensure InputStreams that we open via FileSystem.InputStream(path) are closed. Eagerly closing InputStreams avoids hanging on to file handles until the garbage collector finalizes the InputStream, meaning Bazel on Windows (and other processes) can delete or mutate these files. Hopefully this avoids intermittent file deletion errors that sometimes occur on Windows. See bazelbuild/bazel#5512 RELNOTES: none PiperOrigin-RevId: 203338148

laszlocsomor added type: bug type: process labels Jul 4, 2018

laszlocsomor changed the title ~~Bazel server: ensure InputStreams are closed~~ Bazel server: ensure file streams are closed Jul 4, 2018

laszlocsomor self-assigned this Jul 4, 2018

laszlocsomor added the P1 I'll work on this now. (Assignee required) label Jul 4, 2018

lfpino added the category: misc > misc label Jul 4, 2018

bazel-io pushed a commit that referenced this issue Jul 5, 2018

python tools: ensure files are closed

f11c6bc

Use the "with" statement to open files in various Python scripts used by Bazel, to ensure these files are closed eagerly. See #5512 RELNOTES: none PiperOrigin-RevId: 203346678

laszlocsomor closed this as completed Jul 18, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bazel server: ensure file streams are closed #5512

Bazel server: ensure file streams are closed #5512

laszlocsomor commented Jul 4, 2018

laszlocsomor commented Jul 18, 2018

Bazel server: ensure file streams are closed #5512

Bazel server: ensure file streams are closed #5512

Comments

laszlocsomor commented Jul 4, 2018

Description of the problem / feature request:

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

What's the output of bazel info release?

laszlocsomor commented Jul 18, 2018

What's the output of `bazel info release`?