Invalid UnixFileMode value when using the TarFile API

[WeihanLi]

Description

I'm trying to exact a tar file into a folder, and the tar file may contains folders and files, when I try to use the TarFile.ExtractToDirectory API, I got the error like follows, maybe I'm wrong on using the API

Exception when invoke command System.ArgumentException: Invalid UnixFileMode value. (Parameter 'unixCreateMode')
   at System.IO.Directory.CreateDirectoryCore(String path, UnixFileMode unixCreateMode)
   at System.Formats.Tar.TarHelpers.CreateDirectory(String fullPath, Nullable`1 mode, SortedDictionary`2 pendingModes)
   at System.Formats.Tar.TarEntry.ExtractRelativeToDirectoryAsync(String destinationDirectoryPath, Boolean overwrite, SortedDictionary`2 pendingModes, CancellationToken cancellationToken)
   at System.Formats.Tar.TarFile.ExtractToDirectoryInternalAsync(Stream source, String destinationDirectoryPath, Boolean overwriteFiles, Boolean leaveOpen, CancellationToken cancellationToken)
   at System.Formats.Tar.TarFile.ExtractToDirectoryInternalAsync(Stream source, String destinationDirectoryPath, Boolean overwriteFiles, Boolean leaveOpen, CancellationToken cancellationToken)

Reproduction Steps

var tmpDirPath = Path.Combine(TmpPath, Guid.NewGuid().ToString("N"));
await var fs = File.OpenRead(@"xx.tar.gz");
await using var decompressStream = new GZipStream(fs, CompressionMode.Decompress);
await TarFile.ExtractToDirectoryAsync(decompressStream, tmpDirPath, true);

Expected behavior

Work as normal

Actual behavior

Exception as before

Regression?

No response

Known Workarounds

No response

Configuration

.NET 7 RC2
CentOS 7

Other information

Works well on Windows but not on CentOS

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I'm trying to exact a tar file into a folder, and the tar file may contains folders and files, when I try to use the TarFile.ExtractToDirectory API, I got the error like follows, maybe I'm wrong on using the API

Exception when invoke command System.ArgumentException: Invalid UnixFileMode value. (Parameter 'unixCreateMode')
   at System.IO.Directory.CreateDirectoryCore(String path, UnixFileMode unixCreateMode)
   at System.Formats.Tar.TarHelpers.CreateDirectory(String fullPath, Nullable`1 mode, SortedDictionary`2 pendingModes)
   at System.Formats.Tar.TarEntry.ExtractRelativeToDirectoryAsync(String destinationDirectoryPath, Boolean overwrite, SortedDictionary`2 pendingModes, CancellationToken cancellationToken)
   at System.Formats.Tar.TarFile.ExtractToDirectoryInternalAsync(Stream source, String destinationDirectoryPath, Boolean overwriteFiles, Boolean leaveOpen, CancellationToken cancellationToken)
   at System.Formats.Tar.TarFile.ExtractToDirectoryInternalAsync(Stream source, String destinationDirectoryPath, Boolean overwriteFiles, Boolean leaveOpen, CancellationToken cancellationToken)

Reproduction Steps

var tmpDirPath = Path.Combine(TmpPath, Guid.NewGuid().ToString("N"));
await var fs = File.OpenRead(@"xx.tar.gz");
await using var decompressStream = new GZipStream(fs, CompressionMode.Decompress);
await TarFile.ExtractToDirectoryAsync(decompressStream, tmpDirPath, true);

Expected behavior

Work as normal

Actual behavior

Exception as before

Regression?

No response

Known Workarounds

No response

Configuration

.NET 7 RC2
CentOS 7

Other information

No response

Author:	WeihanLi
Assignees:	-
Labels:	area-System.IO
Milestone:	-

[danmoseley]

Is it possible to share a tar file that reproduces this?

[WeihanLi]

data.tar.gz

Is it possible to share a tar file that reproduces this?

Sure, attached, thanks for the help

[danmoseley]

Do you want to attach a debugger to find out what the unexpected UnixFileMode value is? Perhaps ideally it would be in the message.

[WeihanLi]

Sorry that it's a little hard for me to debug it, I had only Windows to debug, and it works well on Windows. And from the call stack, the unexpected UnixFileMode value may come from the inside since I'm not passing any UnixFileModel value

[danmoseley]

No problem, with the repro file they can easily do it.

[am11]

Here, the directory mode is 16877, but we should just discard the top four bits.

This patch fixes the issue (and passes all the existing tests):

--- a/src/libraries/System.Formats.Tar/src/System/Formats/Tar/TarEntry.cs
+++ b/src/libraries/System.Formats.Tar/src/System/Formats/Tar/TarEntry.cs

         /// <remarks>The value in this field has no effect on Windows platforms.</remarks>
         public UnixFileMode Mode
         {
-            get => (UnixFileMode)_header._mode;
+            get => (UnixFileMode)(_header._mode  & 0xFFF); // mask to keep lower 12 bits and clear out the rest
             set
             {
-                if ((int)value is < 0 or > 4095) // 4095 in decimal is 7777 in octal
+                if ((int)value is < 0 or > 0xFFF) // 0xFFF (4095 in decimal) is 7777 in octal
                 {

[adamsitnik]

@am11 would you like to send a PR?

[am11]

@adamsitnik, we are using the same mask in the writer:

runtime/src/libraries/System.Formats.Tar/src/System/Formats/Tar/TarWriter.Unix.cs

Line 69 in 6dc9a74

entry._header._mode = status.Mode & 4095; // First 12 bits

(term First maybe confusing, but it is the least significant 12 bits)

It seems like other tools are writing mode with higher values, but we are constraining it. So I am not sure if we want to revisit it broadly or conservatively fix this bug. @carlossanlop and/or @tmds might also have opinions on this.

[tmds]

The patch for UnixFileMode property looks good to me.

It seems like other tools are writing mode with higher values, but we are constraining it.

Probably the spec says this should store st_mode which is UnixFileMode + file type (similar to zip ExternalAttributes).
We should update the TarWriter so it matches the spec.

[am11]

@WeihanLi could you provide more info on how did you create this tar.gz file? Please create a smaller one, with one or two directories and a hello world file; max size < 100 kb and if possible, send it via a PR to https://github.com/dotnet/runtime-assets.

I extracted your .tar.gz using command line tar -xzf data.tar.gz and recreated a new tar.gz from the extracted content, then used your code to extract the new tar.gz file. It worked with both GNU as well as BSD flavors. There are number of existing tests extracting such compressed/uncompressed tarballs with directory structure exercising this code.

Also, if we remove GZipStream call from your code and extract the .tar from .tar.gz using gzip -d data.tar.gz, then use your code to extract uncompressed .tar with plain FileStream, it works without throwing any exceptions. The underlying issue seems to be related to interaction between gzip stream and tar reader. @jozkee does it ring a bell?

[jozkee]

The underlying issue seems to be related to interaction between gzip stream and tar reader.

The code splits in some places based on the CanSeekability of a Stream. We recently fixed #74242, that was included on RC2.

[am11]

I am testing with main branch. :)