Downloadable test files

[Bulat-Ziganshin]

Please share in this topic testfiles which can be used to test and compare various deduplication tools. Thanks.

Test files:

• 27 GB Sefaria tarfile compressed to 1.2 GB, requires RAR7 and 32+ GB RAM to decompress
• Original Sefaria data as 7GB zipfile, may change over time

[Sanmayce]

Hi Bulat,
I have many a nifty corpus (1), but there is one standing out, 'Sefaria-master', quite proper it is for [micro-]deduplication benchmarks - because 2 (json vs txt) similar textual corpora are tarred each being 2+GB apart i.e. all the compressors with 2GB window won't be able to see backwards the similar content.
One macro-deduplication alternative could be tarring two identical .iso files 2+GB apart i.e. using a third .iso as a delimiter, for example:
The 'Linux-Sandwich' corpus:
Fedora-Workstation-Live-43-1.6.x86_64.iso (2.6GB)
linuxmint-22.3-mate-64bit.iso (2.9GB)
Fedora-Workstation-Live-43-1.6.x86_64.iso (2.6GB)

The x-factor is that Mint 'Zena' could "contribute" many macro-matches.

[sanmayce@djudjeto ~]$ curl -L -o Sefaria-Export-master.zip https://github.com/Sefaria/Sefaria-Export/archive/refs/heads/master.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:06 --:--:--     0
100 6634M    0 6634M    0     0  4227k      0 --:--:--  0:26:46 --:--:-- 8544k

[sanmayce@djudjeto ~]$ ./7zzs_7z2501-linux-x64 t Sefaria-Export-master.zip 

7-Zip (z) 25.01 (x64) : Copyright (c) 1999-2025 Igor Pavlov : 2025-08-03
 64-bit locale=en_US.UTF-8 Threads:4 OPEN_MAX:1024, ASM

Scanning the drive for archives:
1 file, 6956892931 bytes (6635 MiB)

Testing archive: Sefaria-Export-master.zip
--              
Path = Sefaria-Export-master.zip
Type = zip
Physical Size = 6956892931
64-bit = +
Comment = bb79132716432e0ca5d0e071b4f6d46fa622aa27
Characteristics = Zip64

Everything is Ok                                                                                                       

Folders: 62438
Files: 82694
Size:       27229865002
Compressed: 6956892931

Eugene Roshal set the "tone":
#11 (comment)

For convenience I decompressed the .zip and tarred the folder, then rared the tar with: rarlinux-x64-712 -md28g -mcx

[sanmayce@djudjeto s]$ ls -l Linux_Sandwich/
-rwxr-xr-x 1 sanmayce sanmayce 2742190080 Jan 28 17:53 1_Fedora-Workstation-Live-43-1.6.x86_64.iso
-rwxr-xr-x 1 sanmayce sanmayce 3134275584 Jan 28 18:06 2_linuxmint-22.3-mate-64bit.iso
-rwxr-xr-x 1 sanmayce sanmayce 2742190080 Jan 28 17:53 3_Fedora-Workstation-Live-43-1.6.x86_64.iso

[sanmayce@djudjeto s]$ /bin/time -v ./rarlinux-x64-712 a -md28g -mcx SUPRAPIG_Linux_Sandwich.tar.rar SUPRAPIG_Linux_Sandwich.tar 
[sanmayce@djudjeto s]$ /bin/time -v ./eXdupe-3.0.2_linux_amd64 -k -v3 -x0 SUPRAPIG_Linux_Sandwich.tar SUPRAPIG_Linux_Sandwich.tar.eXdupe0
[sanmayce@djudjeto s]$ /bin/time -v zstd --max SUPRAPIG_Linux_Sandwich.tar -o SUPRAPIG_Linux_Sandwich.tar.zstd157max

-rw-r--r-- 1 sanmayce sanmayce  8618658816 Jan 28 18:24  SUPRAPIG_Linux_Sandwich.tar
-rw-r--r-- 1 sanmayce sanmayce  5394418297 Jan 28 18:46  SUPRAPIG_Linux_Sandwich.tar.eXdupe0
-rw-r--r-- 1 sanmayce sanmayce  5082960117 Jan 28 18:43  SUPRAPIG_Linux_Sandwich.tar.rar
-rw-r--r-- 1 sanmayce sanmayce  7330100642 Jan 28 18:24  SUPRAPIG_Linux_Sandwich.tar.zstd157max

-rw-r--r-- 1 sanmayce sanmayce 27436717568 Jan 28 18:27 'SUPRAPIG_Sefaria-Export-master_(62438-folders_82694-files).tar'
-rw-r--r-- 1 sanmayce sanmayce 15717885224 Jan 28 18:35 'SUPRAPIG_Sefaria-Export-master_(62438-folders_82694-files).tar.eXdupe0'
-rw-r--r-- 1 sanmayce sanmayce  1260838316 Jan 28 19:55 'SUPRAPIG_Sefaria-Export-master_(62438-folders_82694-files).tar.rar'

[sanmayce@djudjeto s]$ b3sum *.rar
9ceeb2a39e11f67630c786eb6acd0658b262b286bee982515edcf96c0dfd22fc  SUPRAPIG_Linux_Sandwich.tar.rar
35740f1e68d24bc891eb5ba4ccc80acd5966dcfe8bf91b4b3632f3ade3d77489  SUPRAPIG_Sefaria-Export-master_(62438-folders_82694-files).tar.rar
[sanmayce@djudjeto s]$ 

The smaller (github complains for 2+GB files) of 2 corpora is downloadable at:
https://github.com/Sanmayce/Zirka/releases/download/v2/SUPRAPIG_Sefaria-Export-master_.62438-folders_82694-files.tar.rar

(1):
There is a house in New Orleans
They call the Rising Sun
And it's been the ruin of many a poor boy
Dear God, I know I was one
/КЪЩАТА НА ИЗГРЯВАЩОТО СЛЪНЦЕ - ДЖЕКИ УЕЛДЖ/

By the way, I have the Zirka_v3 - some original ideas come to play...

[Bulat-Ziganshin]

tarring two identical .iso files 2+GB apart

it's useless for a deduplication program that can follow long LZ matches - the entire second part could be squeezed into a single match

The smaller (github complains for 2+GB files) of 2 corpora is downloadable at

thanks for sharing! especially since it's ready-to-use tar file, allowing fair comparison of different programs (without re-tarring files myself, that will make dedup results depending on the tar utility employed)

Zirka_v3

please make a separate discussion topic here, so we don't mix up everything into one topic