/
compressed_file_format.txt
89 lines (79 loc) · 4.07 KB
/
compressed_file_format.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
###########################################################################################################
# Pcompress File Format. #
###########################################################################################################
Broadly a compressed file consists of a header followed by one or more metadata members which is turn is
followed by one or more compressed chunk data members.
Apart from a standard chunk header, each compressed chunk has an internal format with various metadata
headers of the compression and deduplication algorithms used.
===========================================
File Header
===========================================
8 Bytes - Compression algorithm name
2 Bytes - File format version
2 Bytes - Flags
* * * * * * * * * * * * * * * *
15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
| | | | | | |
| | | | | | `- Simple buffer-level Deduplication on/off
'-------' | | | `----- Fixed Block Deduplication on/off
| | | | Both bits set indicate Global Deduplication.
| | | |
| | | `--------- Solid archive. Entire file compressed in a
| | | single buffer.
| | |
| | `----------------- AES Crypto
| `--------------------- Salsa20 Crypto
|
`------------------------------------- Indicate which data verification checksum
was used.
8 Bytes - Indicated per-thread buffer size
4 Bytes - Compression level
If Encryption Used
-------------------------------------------
4 Bytes - Salt Length
X Bytes - Actual Salt bytes
X Bytes - Nonce: 8 Bytes for AES and 24 Bytes for Salsa20
4 Bytes - Key Length
===========================================
Header Checksum
===========================================
X Bytes - 4 Byte CRC32 without encryption
Header HMAC if encryption enabled. Size of HMAC depends on selected data verification hash.
===========================================
Chunk Header
Each chunk is a single compressed buffer
===========================================
8 Bytes - Compressed Length
X Bytes - Chunk data verification hash (upto 64 bytes) of the original uncompressed and unencrypted
data.
X Bytes - Chunk Header CRC32 for normal compression
Full chunk HMAC, including header, when encrypting. Computation is in this order:
Compression -> Encryption -> HMAC.
1 Byte - Chunk Flags
* * * * * * * *
7 6 5 4 3 2 1 0
| | | | | |
| '-----' | | `- 0 - Uncompressed
| | | | 1 - Compressed
| | | |
| | | `---- 1 - Chunk was Deduped
| | `------- 1 - Chunk was pre-compressed
| |
| | 1 - Lzma (Adaptive Mode)
| | 2 - Bzip2 (Adaptive Mode)
| `---------------- 3 - PPMD (Adaptive Mode)
| 4 - Libbsc (Adaptive Mode)
| 5 - LZ4 (Adaptive Mode)
|
`---------------------- 1 - Chunk size flag (if original chunk is of variable length)
X Bytes - Compressed chunk data
Original uncompressed chunk size can be less than indicated per-thread buffer size. In that
case chunk size bit is set in the flags (as above) and size value is appended after the
compressed chunk data.
-------------------------------------------
8 Bytes - Original uncompressed chunk size
===========================================
File Trailer
===========================================
8 Bytes - Zero bytes indicating zero compressed length
and end of file.