Unsupported compression type 'L4'

[briederer]

I just ran into another error message:

julia> ROOTFile("/hosts/nashome/riederer_bernd/Data/Output/raw-su3-ym-4d-10-2.096680-0.251411-1.987610-adj-200819-085736.root")
ERROR: Unsupported compression type 'L4'
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] UnROOT.Streamers(::IOStream) at /hosts/nashome/riederer_bernd/.julia/packages/UnROOT/KN5MU/src/streamers.jl:85
 [3] ROOTFile(::String) at /hosts/nashome/riederer_bernd/.julia/packages/UnROOT/KN5MU/src/root.jl:39
 [4] top-level scope at none:1

The ROOT-File I am trying to read was created with ROOT v6.14/00 and the settings for File creation were

• Algorithm: Default (which should be LZ4)
• Compreesion-Level: 9

The following lines were obtained from this file using the ROOT-CLI

root [0] 
Attaching file raw-su3-ym-4d-4-9.927979-0.780315-0.171814-adj-200821-082121.root as _file0...
(TFile *) 0x562dada6d350
root [1] _file0->GetCompressionAlgorithm()
(int) 0
root [2] _file0->GetCompressionLevel()
(int) 9
root [3] _file0->GetCompressionSettings()
(int) 9

I guess the program is either catching my case even though it should not be a problem for ZlibDecompressorStream() to read my files. In my opinion this should be the case since all my files with lower Compression-Levels were read without a problem.
If this is not the case however it could maybe be fixed by adding another decompreesion-package like CodecLz4.jl.

[tamasgal]

Yep thanks for the issue! LZ4 is on the todo list, but I have never worked with ROOT files using that compression, so it was not a high priority.

With your file, I can now have a look but in principle it should be fairly easy, since luckily most of the compression codecs in Julia are based on https://github.com/JuliaIO/TranscodingStreams.jl which means that the API is the same!

[tamasgal]

I forgot, could you upload a small sample of that file @bernd1995 ?

[briederer]

Of course.
I added two files with the same content.
test_compression.zip

test_comp.root is the compressed and unreadable version, test_uncomp.root is the readable file.

[tamasgal]

Perfect, thanks!

[briederer]

I tried out my first idea yesterday, to just add the package https://github.com/JuliaIO/CodecLz4.jl and in the function Streamers(io) in streamers.jl change lines 84-88 (streamers.jl#L88) to

if String(compression_header.algo) == "ZL"
    stream = IOBuffer(read(ZlibDecompressorStream(io), tkey.fObjlen))
elseif String(compression_header.algo) == "L4"
    stream = IOBuffer(read(LZ4SafeDecompressorStream(io),tkey.fObjlen))
else
    error("Unsupported compression type '$(String(compression_header.algo))'")
end

However, it seems that there is some different behavior in the read() function for the ::TranscodingStreams.TranscodingStream{LZ4SafeDecompressor,IOStream} datatype than for the ::TranscodingStreams.TranscodingStream{ZlibDecompressor,IOStream} because I get the following error message

f=ROOTFile("/hosts/nashome/riederer_bernd/test_comp.root")
ERROR: LZ4_decompress_safe_continue: Decompression failed.
Stacktrace:
 [1] changemode!(::TranscodingStreams.TranscodingStream{LZ4SafeDecompressor,IOStream}, ::Symbol) at /hosts/nashome/riederer_bernd/.julia/packages/TranscodingStreams/MsN8d/src/stream.jl:717
 [2] callprocess(::TranscodingStreams.TranscodingStream{LZ4SafeDecompressor,IOStream}, ::TranscodingStreams.Buffer, ::TranscodingStreams.Buffer) at /hosts/nashome/riederer_bernd/.julia/packages/TranscodingStreams/MsN8d/src/stream.jl:649
 [3] fillbuffer(::TranscodingStreams.TranscodingStream{LZ4SafeDecompressor,IOStream}; eager::Bool) at /hosts/nashome/riederer_bernd/.julia/packages/TranscodingStreams/MsN8d/src/stream.jl:577
 [4] fillbuffer at /hosts/nashome/riederer_bernd/.julia/packages/TranscodingStreams/MsN8d/src/stream.jl:564 [inlined]
 [5] eof(::TranscodingStreams.TranscodingStream{LZ4SafeDecompressor,IOStream}) at /hosts/nashome/riederer_bernd/.julia/packages/TranscodingStreams/MsN8d/src/stream.jl:188
 [6] readbytes!(::TranscodingStreams.TranscodingStream{LZ4SafeDecompressor,IOStream}, ::Array{UInt8,1}, ::Int32) at /hosts/nashome/riederer_bernd/.julia/packages/TranscodingStreams/MsN8d/src/stream.jl:371
 [7] read(::TranscodingStreams.TranscodingStream{LZ4SafeDecompressor,IOStream}, ::Int32) at ./io.jl:941
 [8] UnROOT.Streamers(::IOStream) at /hosts/nashome/riederer_bernd/.julia/dev/UnROOT/src/streamers.jl:87
 [9] ROOTFile(::String) at /hosts/nashome/riederer_bernd/.julia/dev/UnROOT/src/root.jl:39
 [10] top-level scope at none:1

[tamasgal]

Thanks for the efforts Bernd! Yes I spent two hours yesterday and also could not decompress it.

The LZ4 algorithm requires also the uncompressed size, as far as I understood so I also had to pass tkey.fObjlen to the compressor stream. But still, I get Decompression failed just like you.

I'll dig deeper today. Btw. the compression part needs to be tweaked anyways since currently it's more or less hardcoded to a single algorithm. For this I played around with a new struct to make it easier, but it's not finished yet.

[tamasgal]

Btw. here is the line which I meant: https://github.com/tamasgal/UnROOT.jl/blob/edb08617293e2f248439e40e4952793ada2e16cd/src/streamers.jl#L89

However it does not work yet, as mentioned 😉

[tamasgal]

I guess I will simply dump the raw uncompressed data as a binary file and then fire up the hex editor to inspect it. Additionally I'll also try other LZ4 decompressor libraries (e.g. in Python) to see if it's maybe a problem in CodecLz4.jl. At least uproot is able to read the file...

[tamasgal]

@Moelf figured it out! I think this should work now. @bernd1995 could you give it a try with the lastes master? v0.1.7 should also be ready within the next hour...

See #19

[Moelf]

to be clear, you still can't read data yet, *I think it is because baskets in branch are interlaced with stuff that are not compressed by LZ4 (ROOT's stuff? Tamas can tell me how does that work).

For posterity, LZ4, in block mode, needs to know how long the data is after decompression. AND, there's a 8 bytes checksum (uproot reference) at the beginning of every "block", which is what you guys were missing I think. But also may come to play in branch baskets...

[tamasgal]

Ah ok, sorry for the confusion, at least we have big first leap ;) I'll check it out!

[briederer]

Hey nice work still 😃
I'll try to have a look at it tomorrow but can't promise anything here.

[Moelf]

nvm, I got it working, pushing now!

[tamasgal]

oh wow 😆