ZIP scheme yes or no?

[Oldes]

In my current Rebol version there is built in support for reading/writing ZIP archives using a dedicated codec. There are several usage examples in the codec's test file:

Rebol3/src/tests/units/codecs-test.r3

Lines 253 to 320 in 9c128dc

	if find codecs 'zip [
	v: system/options/log/zip
	system/options/log/zip: 3
	===start-group=== "ZIP codec"
	--test-- "Load ZIP file"
	--assert block? load %units/files/test-lzma.zip
	--assert block? load %units/files/test-stored.zip
	--assert block? load %units/files/test-deflate.zip
	--test-- "Decode ZIP using the codec directly"
	--assert block? data: codecs/zip/decode/only %units/files/test.aar [%classes.jar]
	--assert data/2/3 = 646121705
	--assert block? codecs/zip/decode data/2/2
	--test-- "Decode ZIP using info"
	bin: read %units/files/test-lzma.zip
	--assert block? info: codecs/zip/decode/info bin
	--assert info/1 = %xJSFL.komodoproject
	--assert info/2/1 = 18-Aug-2012/5:20:28
	data: codecs/zip/decompress-file at bin info/2/2 reduce [info/2/5 info/2/3 info/2/4]
	--assert info/2/6 = checksum data 'crc32
	--test-- "Encode ZIP using encode"
	--assert binary? try [bin: encode 'ZIP [
	%empty-folder/ none
	%file-1 "Hello!"
	%file-2 "Hello, Hello, Hello, Hello, Hello!"
	%file-3 #{000102030400010203040001020304}
	%folder/file-4 [1-Jan-2021 "This file is with date!"]
	%file-5 ["Uncompressed" store] ; this file will be included uncompressed
	%file-6 ["This file have a comment" comment: "This file is not important."]
	%file-7 ["File with attributes" att-int: 1 att-ext: 2175008768]
	]]
	data: decode 'ZIP bin
	--assert all [
	data/1 = %empty-folder/
	data/3 = %file-1
	data/4/2 = #{48656C6C6F21}
	data/5 = %file-2
	data/9 = %folder/file-4
	data/10/1 = 1-Jan-2021/0:00
	"Uncompressed" = to string! second select data %file-5
	"This file is not important." = to string! select select data %file-6 'comment
	1 = select select data %file-7 'att-int
	2175008768 = select select data %file-7 'att-ext
	]
	--test-- "Encode ZIP using directory"
	--assert not error? try [save %ico.zip %units/files/ico/]
	data: load %ico.zip
	--assert 30 = length? data ; 14 files and 1 directory
	--assert %ico/ = data/1
	; the order of files is not same across all systems, so next assert is not used
	; --assert %ico/icon_128.png = data/3
	--assert block? select data %ico/icon_128.png
	delete %ico.zip
	--test-- "Encode ZIP using wildcard"
	--assert not error? try [save %temp.zip %units/files/issue-2186*.txt]
	data: load %temp.zip
	--assert 8 = length? data ; 4 files
	--assert %issue-2186-UTF16-BE.txt = data/1
	--assert %issue-2186-UTF16-LE.txt = data/3
	delete %temp.zip
	===end-group===

My question is, if it makes sense to also create a zip scheme? I have a feeling that scheme will not give me so much freedom like the codecs. On the other side, the scheme may make the zip feature more accessible. Is there anybody who wants the scheme and if so, how it should work?

[rebolek]

I believe scheme is a better fit for ZIP (and TAR.GZ) than a codec. Both TAR and ZIP are designed as a supersimple filesystem where you can CRUD (Create, Read, Update, Delete) files, so scheme is the way to go IMO.

[Oldes]

But how it should look like? With real pseudo usage examples!

[Oldes]

The tar codec is not so complete as the zip one...

Rebol3/src/tests/units/codecs-test.r3

Lines 324 to 338 in 9c128dc

	if find codecs 'tar [
	codecs/zip/verbose: 3
	===start-group=== "TAR codec"
	--test-- "Load TAR file"
	--assert block? load %units/files/test.tar
	--test-- "Decode TAR using the codec directly"
	tar-decode: :codecs/tar/decode
	--assert block? data: tar-decode/only %units/files/test.tar %test.txt
	--assert data/2/1 = #{7465737474657374}
	===end-group===
	codecs/tar/verbose: 1
	]

[Oldes]

There is one big advantage of the schemes, that it is higher level and so one could change internal implementation just by changing the scheme's name... if it would done correctly.

[luce80]

Let me try:

>> read/as %file.zip 'zip
== [%file-1 %file-2 %file-3]
>> read/part/as %file.zip %file-1 'zip
== "Hello!"
>> remove %file.zip
== [%file-2 %file-3]
>> remove/part %file.zip %file-3
== [%file-2]
>> write/as %file.zip %file-1 'zip
== [%file-2 %file-1]

But it could conflict with the codec, and I am not sure it is "correct".

[Oldes]

But you are not using any scheme in above code. And such a functionality is already possible with the use of codecs.. like to extract all zip content, you may just do:

>> data: load %/Users/oldes/Downloads/imreadmulti_bug.zip 
 [ZIP] Decode ZIP data ( 965 bytes )
 [ZIP] Extracting:  imreadmulti_bug/  bytes: 0 -> 0
 [ZIP] Extracting:  imreadmulti_bug/openmulti.py  bytes: 168 -> 242
 [ZIP] Extracting:  imreadmulti_bug/2page.tiff  bytes: 251 -> 4004
== [
    %imreadmulti_bug/
    [16-Sep-2017/11:44:58 none 0 extra: #{55540500039AF2BC5975780B000104E803000004E8030000} att-ext: 1106051088]
    %imreadmulti_bug/openmulti.py
    [26-Aug-2017/13:13:38 #{
696D706F7274206376320A696D706F7274206E756D7079206173206E700A696D
706F727420707072696E740A0A6D796C697374203D205B5D0A6C6F6164656420
3D206376322E696D726561646D756C7469286D617473203D206D796C6973742C
2066696C656E616D65203D202232706167652E74696666222C20666C61677320
3D206376322E494D524541445F414E59434F4C4F...

[Oldes]

The use of scheme is more like a virtual filesystem, as @rebolek said... where you don't want to extract every file from a large archive.

It would be more like:

archive: open zip://path/to/archive.zip
;; now what to do with the opened archive?

There are many actions which are possible on the port. The question is, which to support and how. For example I can imagine to get single record using...

data: pick archive %imreadmulti_bug/openmulti.py
;;or
data: read zip://path/to/archive.zip?imreadmulti_bug/openmulti.py

But there is still so many questions... like :

• Should there be support for traversing?
• What should be different between pick and select
• What if we query a directory record?
  etc.

[luce80]

such a functionality is already possible with the use of codecs

Sorry but I was not showing the "functionality" but its different "grammar" in a non-codec approch. I clearly misunderstood your question and didn't understood you were interested in its low-level aspects. That's too deep for me ;) . But I think that if you are seeing it as a (virtual) filesystem you should treat it as such as much as possible. The difficult part is to let the user manipulate the archive also as a "normal" file but if he/she then wants to use the archive functionality he will use your low-level scheme (or the codec if that will remain).