Support long file/path names for TarWriter/TarReader

Currently TarWriter limits file/path names to 256 (or so) characters.

However, for example this test file can contain files with path names exceeding 256 characters. That file was made by GNU tar 1.28.

Would you please support long file name (path name) for TarWriter (and possibly TarReader)?

Thanks,

[djberg96]

I'm not actually sure why that check is in there unless it's some attempt to comply with an old POSIX spec.

[drbrain]

Is there a specification for modern tar files?

Also, old rubygems won't be able to install tar files if we make this upgrade.

[drbrain]

I suppose we could add support and a warning upon packaging when using long file names, but we need a link to the spec to implement support.

[djberg96]

@drbrain This is the best I could find:

https://www.gnu.org/software/tar/manual/html_section/tar_68.html
https://en.wikipedia.org/wiki/Tar_(computing)

I think the 256 path limit is POSIX-1988 or earlier that hung around in some implementations. I vaguely recall Solaris tar suffering from this issue and having to replace it with GNU tar. From what I can gather, most Unixy implementations set path max at 4k now, or have no limit. Now if we're talking about a filename limit, that I think it still generally set to 256 on Unixy platforms, but it depends on the filesystem type.

Anyway, what happens if we just remove that check?

[drbrain]

New tar stores the filename differently, the 256 bytes comes from the sizes of the name and prefix entries in the tar header

In order to support longer filenames we need to implement this ‼ thing, it seems:

$ hexdump -C test.tar | more
00000000  21 21 30 2f 00 00 00 00  00 00 00 00 00 00 00 00  |!!0/............|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000060  00 00 00 00 30 30 30 30  37 30 30 00 30 30 30 31  |....0000700.0001|
00000070  37 35 30 00 30 30 30 31  37 35 30 00 30 30 30 30  |750.0001750.0000|
00000080  30 30 30 30 30 30 30 00  31 32 36 31 37 33 32 33  |0000000.12617323|
00000090  33 36 30 00 30 30 37 30  35 34 00 20 35 00 00 00  |360.007054. 5...|
000000a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000100  00 75 73 74 61 72 20 20  00 68 62 00 00 00 00 00  |.ustar  .hb.....|
00000110  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000120  00 00 00 00 00 00 00 00  00 68 62 00 00 00 00 00  |.........hb.....|
00000130  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200  21 21 30 2f 31 2f 00 00  00 00 00 00 00 00 00 00  |!!0/1/..........|
00000210  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000260  00 00 00 00 30 30 30 30  37 30 30 00 30 30 30 31  |....0000700.0001|
00000270  37 35 30 00 30 30 30 31  37 35 30 00 30 30 30 30  |750.0001750.0000|
00000280  30 30 30 30 30 30 30 00  31 32 36 31 37 33 32 33  |0000000.12617323|
00000290  33 36 30 00 30 30 37 32  31 34 00 20 35 00 00 00  |360.007214. 5...|
000002a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000300  00 75 73 74 61 72 20 20  00 68 62 00 00 00 00 00  |.ustar  .hb.....|
00000310  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000320  00 00 00 00 00 00 00 00  00 68 62 00 00 00 00 00  |.........hb.....|
00000330  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000400  21 21 30 2f 31 2f 32 32  2f 00 00 00 00 00 00 00  |!!0/1/22/.......|
00000410  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000460  00 00 00 00 30 30 30 30  37 30 30 00 30 30 30 31  |....0000700.0001|
00000470  37 35 30 00 30 30 30 31  37 35 30 00 30 30 30 30  |750.0001750.0000|
00000480  30 30 30 30 30 30 30 00  31 32 36 31 37 33 32 33  |0000000.12617323|
00000490  33 36 30 00 30 30 37 34  33 37 00 20 35 00 00 00  |360.007437. 5...|
000004a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000500  00 75 73 74 61 72 20 20  00 68 62 00 00 00 00 00  |.ustar  .hb.....|
00000510  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000520  00 00 00 00 00 00 00 00  00 68 62 00 00 00 00 00  |.........hb.....|
00000530  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000600  21 21 30 2f 31 2f 32 32  2f 33 33 33 2f 00 00 00  |!!0/1/22/333/...|
00000610  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*

Thank you for accepting this request, that's great :-)

I was going to find tar format from Wikipedia and GNU tar project, like Daniel did.

That folder name !!0 was my mistake, it has no special meaning. I often prefix folder/file names with !! and tell Git to ignore those names.

For backward compatibility, I think you can add a flag like use_long_names, default to false. That way people won't need to update code...

I used monkey patching to remove the limit check, it produced a broken file.

This is code:

#!/usr/bin/env ruby

require 'rubygems/package'

IO_BUFFER = 1024 * 64

class Gem::Package::TarWriter
    def split_name(name) # :nodoc:
        if name.bytesize <= 100 then
            prefix = ""
        else
            parts = name.split(/\//)
            newname = parts.pop
            nxt = ""

            loop do
                nxt = parts.pop
                break if newname.bytesize + 1 + nxt.bytesize > 100
                newname = nxt + "/" + newname
            end

            prefix = (parts + [nxt]).join "/"
            name = newname
        end

        return name, prefix
    end
end

if __FILE__ == $0
    tar_reader, tar_writer = IO.pipe

    # tar handler
    tar_handler_thread = Thread.new do
        File.open('test2.tar', 'wb') do |io|
            while data = tar_reader.read(IO_BUFFER) do
                io.write data
            end
        end
    end

    Gem::Package::TarWriter.new(tar_writer) do |writer|
        original_size = 0

        0.upto(50) do |index|
            path = '!!0'
            1.upto(index) { |i| path = File.join(path, i.to_s * i) }

            test = "Hellboy III, please!\n"
            writer.add_file_simple(File.join(path, 'test.txt'), 0644, test.bytesize) do |io|
                io.write test
                original_size += test.bytesize
            end
        end

        puts " > Total size = #{original_size} bytes"
    end

    tar_writer.close
    tar_handler_thread.join
    tar_reader.close
end

[drbrain]

Yes, it would, because you're overflowing the 512 byte tar header. Per the hexdump, modern tar stores the filename in a !! section, but without a specification its too hard to know if it is implemented correctly.