Recompression output (input -> decompress -> compress -> output) is different from input

[protyposis]

There seems to be an issue somewhere in the decompression and/or compression algorithms, because unmodified recompressed firmware data is different from the original firmware data. As long as this issue isn't solved, I do not recommend anyone to try and flash a recompressed/repacked firmware, it will most likely fail and possibly damage the camera.

This can be tested using the npm run test <firmwarefile> command, which will unpack and decompress the firmware sections, recompress them, and compare differences. It will output differences like this:

// lookup index data decompression vs. compression
// left: compressed input byte index, decompressed output byte index, lookup index, lookup length, lookup buffer size
// right:  compressed output byte index, decompressed input byte index, lookup index, lookup length, lookup buffer size
lookup mismatch: 363,427,137,3,409 <-> 363,427,199,3,409
lookup mismatch: 373,438,136,3,420 <-> 373,438,198,3,420
lookup mismatch: 588,864,138,3,846 <-> 588,864,200,3,846

Some information about how the compression algorithm works can be found here and in a reverse engineering SE question. The first line tells that the original firmware file has a lookup encoded with 3 bytes length at lookup buffer index 137, whereas the recompression selected lookup buffer index 199. Inspecting the lookup buffer shows that the 3 bytes can be found at both locations (actually it can be found in even more locations), but the recompression selects 199 because it's nearest to the tail of the circular lookup buffer where the reverse lookup search begins.

In most cases taking the nearest lookup index from the tail of the lookup buffer yields the correct lookup index, it's just in a few rare cases where it is wrong. This probably means that

• either there is a bug in the decompression algorithm and the first decompression is wrong, and the lookup data in the case above should not be located at index 199 but only at 137 which the compression algorithm would then select correctly
• or there's some weird deliberate optimization or undeliberate lookup index picking strategy implemented in the original algorithm that I cannot figure out

[skytina]

Hi, @protyposis , i do the same test on v3.2-cn firmware.
Recompression output is same as input,which mean what you think is correct!

Test1. npm run test

The Output of command npm run test like this
`
npm run test ~/xx/Iot_Devices_Vulns/camera/xiaoyi/3.2-cn/firmware.bin

yi-mirrorless-firmware-tools@0.1.0 test /home/skytina/GitProject/IotUnpacker/yi-mirrorless-firmware-tools
node index.js test "/home/skytina/xx/Iot_Devices_Vulns/camera/xiaoyi/3.2-cn/firmware.bin"

----- Section 0 -----
Raw header string: LENGTH=7370752 C59Y1 VER=M1CN DVR=Ver1.42 SUM=937552776 ND1 IPL PTBL
Parsed header: { sectionLength: 7370752,
deviceId: 'C59Y1',
deviceVersion: 'M1CN',
dvr: 'Ver1.42',
sectionSum: 937552776,
followingSectionIds: [ 'ND1', 'IPL', 'PTBL' ] }
# WARNING ###########################################################
# Cannot identify firmware: unknown device version M1CN #
# Please open an issue to get this version added: #
# https://github.com/protyposis/yi-mirrorless-firmware-tools/issues #
#####################################################################
Section read ok (7370752 bytes)
Checksum test ok (937552776)
Section 0
----- Section 1 -----
Raw header string: ND1 LENGTH=4197888 C59Y1 VER=M1CN DVR=Ver1.42 SUM=299791776 OFFSET=23068672
Parsed header: { sectionId: 'ND1',
sectionLength: 4197888,
deviceId: 'C59Y1',
deviceVersion: 'M1CN',
dvr: 'Ver1.42',
sectionSum: 299791776,
sectionOffset: 23068672 }
Section read ok (4197888 bytes)
Checksum test ok (299791776)
Section 1
----- Section 2 -----
Raw header string: IPL LENGTH=131072 C59Y1 VER=M1CN DVR=Ver1.42 SUM=5714438
Parsed header: { sectionId: 'IPL',
sectionLength: 131072,
deviceId: 'C59Y1',
deviceVersion: 'M1CN',
dvr: 'Ver1.42',
sectionSum: 5714438 }
Section read ok (131072 bytes)
Checksum test ok (5714438)
Section 2
----- Section 3 -----
Raw header string: PTBL LENGTH=4096 C59Y1 VER=M1CN SUM=5181
Parsed header: { sectionId: 'PTBL',
sectionLength: 4096,
deviceId: 'C59Y1',
deviceVersion: 'M1CN',
sectionSum: 5181 }
Section read ok (4096 bytes)
Checksum test ok (5181)
Section 3
EOF
`

Test2. check the md5 after the unpack and repack manually

check the md5 after the unpack and repack manually, the md5 hash between repack and orignal is same

ls -l firmware.bin firmware.bin.bak
-rw-r--r-- 1 skytina skytina 11704832 8月 27 2018 firmware.bin
-rw-r--r-- 1 skytina skytina 11704832 3月 16 15:08 firmware.bin.bak
->md5sum firmware.bin
6b2a54cf75d394c6ac29ff4cd34da00f firmware.bin
->md5sum firmware.bin.bak
6b2a54cf75d394c6ac29ff4cd34da00f firmware.bin.bak

---

I'm really appreciated for what you do,i hope that those informations is useful for you.(Sorry for my poor english)

[protyposis]

@skytina thank you for your feedback! I also tested 3.2-int and can confirm that repacking works correctly.

Including the report in #2 (comment), we have 3 confirmations that repacking of v3.2 works correctly.

Please be advised that repacking of lower versions still does not work correctly. I have a report that flashing a repacked 3.1 firmware with the differences explained above resulted in a broken camera (does not turn on).