request to search from 7z archive

[charulix]

would like to request a feature to read from 7zip archive on search

Search the contents of archives (

Thanks

[genivia-inc]

Saw that one coming from a mile away 😄

I had this on my TODO list. It is nice to have, BUT sadly this will not be technically easy to add to ugrep or may not even be legally possible.

Firstly, 7zip has no usable C/C++ library to link ugrep with. Also 7zip is rather complex in its internals, which seems to me has to be replicated in ugrep accurately as roughly described here: https://www.romvault.com/Understanding7z.pdf As you can see, it would be like writing the source code for 7zip from scratch. It is not as simple as forking 7z -x to extract files.

Secondly, we cannot use any of the 7zip source code for ugrep directly, because 7zip is LGPL and cannot be used with a BSD3 project like ugrep. The other (de)compression libraries are linked (not compiled) with ugrep, so these do not pose a legal problem from a licensing point of view.

To keep ugrep clean and unencumbered by licensing issues, I wrote my own tar, zip, pax and cpio unarchivers from scratch in C++ that call low-level external decompression functions of libraries (zlib,bzip2,lzma,lz4,zstd) linked with ugrep.

[Bladehawke]

As a follow on, and noting that I am not well versed on the complexities of software licensing, could you not link against 7z.dll (or 7za.dll) noting that your project uses it and that it (not your project) is LGPL? I guess my thought comes from https://www.7-zip.org/faq.html#developer_faq

[charulix]

no worries, not sure on how to use the dll. not using ugrep on my project yet. running all in pure batch at the moment. much easier to maintain. thanks

…

On Mon, Jun 13, 2022 at 1:56 PM Xander ***@***.***> wrote: As a follow on, and noting that I am not well versed on the complexities of software licensing, could you not link against 7z.dll (or 7za.dll) noting that your project uses it and that it (not your project) is LGPL? I guess my thought comes from https://www.7-zip.org/faq.html#developer_faq — Reply to this email directly, view it on GitHub <#185 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUYX4RL3PQQH4HVUX5QYT6TVO3ESJANCNFSM5L25CREA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[genivia-inc]

The LZMA SDK includes a CPP/7zip/Archive/7z directory with the C++ source code to extract files, which is a good starting point to look into this, because this code is placed in the public domain. It includes the C++ source code for .7z compression and decompression, albeit a reduced version, so not everything that 7-zip supports may work.

[genivia-inc]

So it turns out the documentation is practically non-existent on how to use this 7-zip LZMA SDK's API to decompress archives. No source code comments either. That's awful. It may be free, but comes at a high cost (reverse engineer the API + dev time + debugging/testing).

At a minimum we need to use the lower-level API to decompress .7z archives incrementally in memory, fetch the compressed path names (to display) and decompress their content in memory to send to the search engine.

[hdatma]

The following works well enough:

ugrep --filter='7z:7z x -so %' pattern filename.7z

The trick can be included in the default configuration:

filter=7z:7z x -so %

[genivia-inc]

The following works well enough:

ugrep --filter='7z:7z x -so %' pattern filename.7z

Also specify option -W to search 7zip that may contain binary files. This option prevents bailing out with a "binary file matches" warning and instead shows binary matches in hex.

[charulix]

thank you

[genivia-inc]

I'm reopening this request.

Upon closer inspection of the LZMA SDK it appears there is everything that I need to extract 7zip compressed files in memory to search. I was wrong about licensing. LZMA SDK is placed in the public domain. Part of the LZMA SDK source code files have to be copied in order to compile a decompressor. I will use the LZMA SDK C source code files, which produce smaller object files to link with ugrep.

I will give this a go in the coming days.

[charulix]

thank you so much for the update

…

On Tue, 26 Dec 2023, 4:43 am Dr. Robert van Engelen, < ***@***.***> wrote: I'm reopening this request. Upon closer inspection of the LZMA SDK it appears there is everything that I need to extract 7zip compressed files in memory to search. I was wrong about licensing. LZMA SDK is placed in the public domain. Part of the LZMA SDK source code files have to be copied in order to compile a decompressor. I will use the LZMA SDK C source code files, which produce smaller object files to link with ugrep. I will give this a go in the coming days. — Reply to this email directly, view it on GitHub <#185 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUYX4RPAMJGRMBQRUHHN5NTYLHQP7AVCNFSM5L25CREKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBWHEYTCMRUG4YA> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>