New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8KB of data garbled at 2TB boundary #51

Closed
ivan opened this Issue Nov 22, 2015 · 13 comments

Comments

Projects
None yet
2 participants
@ivan

ivan commented Nov 22, 2015

First, thanks for dislocker, it's great to be able to read BitLocker disks on Linux!

I have a 4TB BitLocker volume that I created on Windows 10. I checksummed 1.8TB of files on this volume, and found a single file that dislocker appears to be reading incorrectly. Specifically, two 4KB blocks in the file appear to be garbled. Windows 10 reads the file correctly (tested twice on different weeks) while dislocker reads the file incorrectly (tested twice across two physical-power-off events).

I built dislocker@develop 564420c on Ubuntu 15.10 (64-bit).

With dislocker:

# cd /home/dislocker/dislocker/src

# ./dislocker-fuse --readonly --verbosity --user-password -V /dev/sda4 /mnt/bitlocker-C
Enter the user password:

# ls -l /mnt/bitlocker-C/dislocker-file 
-r--r--r-- 1 root root 4,000,191,610,368 1970-01-01 00:00 /mnt/bitlocker-C/dislocker-file

# mount -t ntfs-3g -o ro,uid=1000,gid=1000,umask=077 /mnt/bitlocker-C/dislocker-file /mnt/C

# cd /mnt/C/YouTube/UCNHAsCPp3tfbV3O3t7RxYMA

# ls -l 'Forbush, Colombatto, Shuen, Polleit, Huelsmann, Discusssion, Q & A (PFS 2015)-v5zr8s0Zmn8.webm'
-rwx------ 2 at at 605,594,334 2015-10-13 08:00 Forbush, Colombatto, Shuen, Polleit, Huelsmann, Discusssion, Q & A (PFS 2015)-v5zr8s0Zmn8.webm*

# md5sum 'Forbush, Colombatto, Shuen, Polleit, Huelsmann, Discusssion, Q & A (PFS 2015)-v5zr8s0Zmn8.webm' 
f943abbd99343db0d4e384524e8d588a  Forbush, Colombatto, Shuen, Polleit, Huelsmann, Discusssion, Q & A (PFS 2015)-v5zr8s0Zmn8.webm

# ffmpeg -v error -i 'Forbush, Colombatto, Shuen, Polleit, Huelsmann, Discusssion, Q & A (PFS 2015)-v5zr8s0Zmn8.webm' -f null - 
[matroska,webm @ 0x13c5c00] Invalid EBML number size tag 0x05 at pos 572409231 (0x221e458f)

I booted into Windows 10 after seeing the incorrect hash f943abbd99343db0d4e384524e8d588a and confirmed that Windows 10 still saw the correct hash 61ac59d3b615cf191b22351b706d7db4.

On a known-good copy of the data created on Windows 10, copying into a Linux Samba share:

# md5sum 'Forbush, Colombatto, Shuen, Polleit, Huelsmann, Discusssion, Q & A (PFS 2015)-v5zr8s0Zmn8.webm'
61ac59d3b615cf191b22351b706d7db4  Forbush, Colombatto, Shuen, Polleit, Huelsmann, Discusssion, Q & A (PFS 2015)-v5zr8s0Zmn8.webm

# ffmpeg -v error -i 'Forbush, Colombatto, Shuen, Polleit, Huelsmann, Discusssion, Q & A (PFS 2015)-v5zr8s0Zmn8.webm' -f null -

(no errors)

Binary-diffing the file shows these differences, indicating that two 4KB blocks are garbled.

strace'ing the mount.ntfs-3g process and then running this program to read the garbled block:

python -c "f=open('Forbush, Colombatto, Shuen, Polleit, Huelsmann, Discusssion, Q & A (PFS 2015)-v5zr8s0Zmn8.webm', 'rb');f.seek(568303648);print hash(f.read(8*1024))"

shows

read(4, "8\0\0\0\3\0\0\0\357\22\0\0\0\0\0\0\3\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 56
writev(4, [{"x\0\0\0\0\0\0\0\357\22\0\0\0\0\0\0", 16}, {"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\232\372;\0\0\0\0\0\0\0\2\0\0\0\0\0"..., 104}], 2) = 120
read(4, "\207\0\0\0\1\0\0\0\360\22\0\0\0\0\0\0\3\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 135
writev(4, [{"\220\0\0\0\0\0\0\0\360\22\0\0\0\0\0\0", 16}, {"\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 128}], 2) = 144
read(4, "8\0\0\0\3\0\0\0\361\22\0\0\0\0\0\0\4\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 56
writev(4, [{"x\0\0\0\0\0\0\0\361\22\0\0\0\0\0\0", 16}, {"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0Y\373;\0\0\0\0\0\336\242\30$\0\0\0\0"..., 104}], 2) = 120
read(4, "0\0\0\0\16\0\0\0\362\22\0\0\0\0\0\0\4\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 48
writev(4, [{" \0\0\0\0\0\0\0\362\22\0\0\0\0\0\0", 16}, {"\0\0\0\0\0\0\0\0\2\0\0\0\0\0\0\0", 16}], 2) = 32
read(4, "8\0\0\0\3\0\0\0\363\22\0\0\0\0\0\0\4\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 56
writev(4, [{"x\0\0\0\0\0\0\0\363\22\0\0\0\0\0\0", 16}, {"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0Y\373;\0\0\0\0\0\336\242\30$\0\0\0\0"..., 104}], 2) = 120
read(4, "8\0\0\0\3\0\0\0\364\22\0\0\0\0\0\0\4\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 56
writev(4, [{"x\0\0\0\0\0\0\0\364\22\0\0\0\0\0\0", 16}, {"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0Y\373;\0\0\0\0\0\336\242\30$\0\0\0\0"..., 104}], 2) = 120
read(4, "@\0\0\0\22\0\0\0\365\22\0\0\0\0\0\0\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 135168) = 64
writev(4, [{"\20\0\0\0\0\0\0\0\365\22\0\0\0\0\0\0", 16}], 1) = 16
read(4, "8\0\0\0\3\0\0\0\366\22\0\0\0\0\0\0\3\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 56
writev(4, [{"x\0\0\0\0\0\0\0\366\22\0\0\0\0\0\0", 16}, {"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\232\372;\0\0\0\0\0\0\0\2\0\0\0\0\0"..., 104}], 2) = 120
read(4, "-\0\0\0\1\0\0\0\367\22\0\0\0\0\0\0\3\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
pread(3, "INDX(\0\t\0h\0019\366\10\0\0\0\3\0\0\0\0\0\0\0(\0\0\0\0\10\0\0"..., 4096, 2905324924928) = 4096
pread(3, "INDX(\0\t\0\204\0051\366\10\0\0\0\0\0\0\0\0\0\0\0(\0\0\0\250\7\0\0"..., 4096, 2778237616128) = 4096
writev(4, [{"\20\0\0\0\376\377\377\377\367\22\0\0\0\0\0\0", 16}], 1) = 16

Is there any other debugging information that might be useful?

@ivan

This comment has been minimized.

Show comment
Hide comment
@ivan

ivan Nov 22, 2015

This file is not compressed and sysinternals Contig reported that it is stored in one fragment.

ivan commented Nov 22, 2015

This file is not compressed and sysinternals Contig reported that it is stored in one fragment.

@Aorimn

This comment has been minimized.

Show comment
Hide comment
@Aorimn

Aorimn Nov 23, 2015

Owner

Is it possible for you to run a dislocker-metadata on the BitLockered partition? It would help to know if these blocks are near a special area.

In the same spirit, are these two blocks at the end of the file? Or are they in the middle?

Owner

Aorimn commented Nov 23, 2015

Is it possible for you to run a dislocker-metadata on the BitLockered partition? It would help to know if these blocks are near a special area.

In the same spirit, are these two blocks at the end of the file? Or are they in the middle?

@ivan

This comment has been minimized.

Show comment
Hide comment
@ivan

ivan Nov 23, 2015

# ./dislocker-metadata -V /dev/sda4
Mon Nov 23 17:40:25 2015 [INFO] dislocker by Romain Coltel, v0.4.1 (compiled for Linux/x86_64)
Mon Nov 23 17:40:25 2015 [INFO] Compiled version: develop:bdeda46
Mon Nov 23 17:40:25 2015 [INFO] Volume GUID (INFORMATION OFFSET) supported
Mon Nov 23 17:40:25 2015 [INFO] BitLocker metadata found and parsed.
Mon Nov 23 17:40:25 2015 [INFO] =====[ Volume header informations ]=====
Mon Nov 23 17:40:25 2015 [INFO]   Signature: '-FVE-FS-'
Mon Nov 23 17:40:25 2015 [INFO]   Sector size: 0x0200 (512) bytes
Mon Nov 23 17:40:25 2015 [INFO]   Sector per cluster: 0x08 (8) bytes
Mon Nov 23 17:40:25 2015 [INFO]   Reserved clusters: 0x0000 (0) bytes
Mon Nov 23 17:40:25 2015 [INFO]   Fat count: 0x00 (0) bytes
Mon Nov 23 17:40:25 2015 [INFO]   Root entries: 0x0000 (0) bytes
Mon Nov 23 17:40:25 2015 [INFO]   Number of sectors (16 bits): 0x0000 (0) bytes
Mon Nov 23 17:40:25 2015 [INFO]   Media descriptor: 0xf8 (248) bytes
Mon Nov 23 17:40:25 2015 [INFO]   Sectors per fat: 0x0000 (0) bytes
Mon Nov 23 17:40:25 2015 [INFO]   Hidden sectors: 0x0011b800 (1161216) bytes
Mon Nov 23 17:40:25 2015 [INFO]   Number of sectors (32 bits): 0x00000000 (0) bytes
Mon Nov 23 17:40:25 2015 [INFO]   Number of sectors (64 bits): 0x0000000000000000 (0) bytes
Mon Nov 23 17:40:25 2015 [INFO]   MFT start cluster: 0x0000000000060001 (393217) bytes
Mon Nov 23 17:40:25 2015 [INFO]   Metadata Lcn: 0x0000000000000000 (0) bytes
Mon Nov 23 17:40:25 2015 [INFO]   Volume GUID: [...]
Mon Nov 23 17:40:25 2015 [INFO]   First metadata header offset:  0x000000001f200000
Mon Nov 23 17:40:25 2015 [INFO]   Second metadata header offset: 0x000000005f200000
Mon Nov 23 17:40:25 2015 [INFO]   Third metadata header offset:  0x000000009f7fc000
Mon Nov 23 17:40:25 2015 [INFO]   Boot Partition Identifier: '0xaa55'
Mon Nov 23 17:40:25 2015 [INFO] ========================================
Mon Nov 23 17:40:25 2015 [INFO] 
Mon Nov 23 17:40:25 2015 [INFO] =====================[ BitLocker information structure ]=====================
Mon Nov 23 17:40:25 2015 [INFO]   Signature: '-FVE-FS-'
Mon Nov 23 17:40:25 2015 [INFO]   Total Size: 0x0380 (896) bytes (including signature and data)
Mon Nov 23 17:40:25 2015 [INFO]   Version: 2
Mon Nov 23 17:40:25 2015 [INFO]   Current state: ENCRYPTED (4)
Mon Nov 23 17:40:25 2015 [INFO]   Next state: ENCRYPTED (4)
Mon Nov 23 17:40:25 2015 [INFO]   Encrypted volume size: 4000191610880 bytes (0x3a35e000000), ~3814880 MB
Mon Nov 23 17:40:25 2015 [INFO]   Size of convertion region: 0 (0)
Mon Nov 23 17:40:25 2015 [INFO]   Number of boot sectors backuped: 16 sectors (0x10)
Mon Nov 23 17:40:25 2015 [INFO]   First metadata header offset:  0x1f200000
Mon Nov 23 17:40:25 2015 [INFO]   Second metadata header offset: 0x5f200000
Mon Nov 23 17:40:25 2015 [INFO]   Third metadata header offset:  0x9f7fc000
Mon Nov 23 17:40:25 2015 [INFO]   Boot sectors backup address:   0x1f210000
Mon Nov 23 17:40:25 2015 [INFO]   ----------------------------{ Dataset header }----------------------------
Mon Nov 23 17:40:25 2015 [INFO]     Dataset size: 0x0000033a (826) bytes (including data)
Mon Nov 23 17:40:25 2015 [INFO]     Unknown data: 0x00000001 (always 0x00000001)
Mon Nov 23 17:40:25 2015 [INFO]     Dataset header size: 0x00000030 (always 0x00000030)
Mon Nov 23 17:40:25 2015 [INFO]     Dataset copy size: 0x0000033a (826) bytes
Mon Nov 23 17:40:25 2015 [INFO]     Dataset GUID: [...]
Mon Nov 23 17:40:25 2015 [INFO]     Next counter: 10
Mon Nov 23 17:40:25 2015 [INFO]     Encryption Type: AES-128-NODIFFUSER (0x8002)
Mon Nov 23 17:40:25 2015 [INFO]     Epoch Timestamp: 1438841786 sec, that to say Thu Aug  6 06:16:26 2015
Mon Nov 23 17:40:25 2015 [INFO]   --------------------------------------------------------------------------
Mon Nov 23 17:40:25 2015 [INFO] =============================================================================
Mon Nov 23 17:40:25 2015 [INFO] 
Mon Nov 23 17:40:25 2015 [INFO] 
Mon Nov 23 17:40:25 2015 [INFO] =======[ Datum n°1 informations ]=======
[...]

Do you think it might have run into a BitLocker metadata header? I see those are also 8192 bytes.

The corruption is ~568,303,648 bytes into a 605,594,334 byte file.

ivan commented Nov 23, 2015

# ./dislocker-metadata -V /dev/sda4
Mon Nov 23 17:40:25 2015 [INFO] dislocker by Romain Coltel, v0.4.1 (compiled for Linux/x86_64)
Mon Nov 23 17:40:25 2015 [INFO] Compiled version: develop:bdeda46
Mon Nov 23 17:40:25 2015 [INFO] Volume GUID (INFORMATION OFFSET) supported
Mon Nov 23 17:40:25 2015 [INFO] BitLocker metadata found and parsed.
Mon Nov 23 17:40:25 2015 [INFO] =====[ Volume header informations ]=====
Mon Nov 23 17:40:25 2015 [INFO]   Signature: '-FVE-FS-'
Mon Nov 23 17:40:25 2015 [INFO]   Sector size: 0x0200 (512) bytes
Mon Nov 23 17:40:25 2015 [INFO]   Sector per cluster: 0x08 (8) bytes
Mon Nov 23 17:40:25 2015 [INFO]   Reserved clusters: 0x0000 (0) bytes
Mon Nov 23 17:40:25 2015 [INFO]   Fat count: 0x00 (0) bytes
Mon Nov 23 17:40:25 2015 [INFO]   Root entries: 0x0000 (0) bytes
Mon Nov 23 17:40:25 2015 [INFO]   Number of sectors (16 bits): 0x0000 (0) bytes
Mon Nov 23 17:40:25 2015 [INFO]   Media descriptor: 0xf8 (248) bytes
Mon Nov 23 17:40:25 2015 [INFO]   Sectors per fat: 0x0000 (0) bytes
Mon Nov 23 17:40:25 2015 [INFO]   Hidden sectors: 0x0011b800 (1161216) bytes
Mon Nov 23 17:40:25 2015 [INFO]   Number of sectors (32 bits): 0x00000000 (0) bytes
Mon Nov 23 17:40:25 2015 [INFO]   Number of sectors (64 bits): 0x0000000000000000 (0) bytes
Mon Nov 23 17:40:25 2015 [INFO]   MFT start cluster: 0x0000000000060001 (393217) bytes
Mon Nov 23 17:40:25 2015 [INFO]   Metadata Lcn: 0x0000000000000000 (0) bytes
Mon Nov 23 17:40:25 2015 [INFO]   Volume GUID: [...]
Mon Nov 23 17:40:25 2015 [INFO]   First metadata header offset:  0x000000001f200000
Mon Nov 23 17:40:25 2015 [INFO]   Second metadata header offset: 0x000000005f200000
Mon Nov 23 17:40:25 2015 [INFO]   Third metadata header offset:  0x000000009f7fc000
Mon Nov 23 17:40:25 2015 [INFO]   Boot Partition Identifier: '0xaa55'
Mon Nov 23 17:40:25 2015 [INFO] ========================================
Mon Nov 23 17:40:25 2015 [INFO] 
Mon Nov 23 17:40:25 2015 [INFO] =====================[ BitLocker information structure ]=====================
Mon Nov 23 17:40:25 2015 [INFO]   Signature: '-FVE-FS-'
Mon Nov 23 17:40:25 2015 [INFO]   Total Size: 0x0380 (896) bytes (including signature and data)
Mon Nov 23 17:40:25 2015 [INFO]   Version: 2
Mon Nov 23 17:40:25 2015 [INFO]   Current state: ENCRYPTED (4)
Mon Nov 23 17:40:25 2015 [INFO]   Next state: ENCRYPTED (4)
Mon Nov 23 17:40:25 2015 [INFO]   Encrypted volume size: 4000191610880 bytes (0x3a35e000000), ~3814880 MB
Mon Nov 23 17:40:25 2015 [INFO]   Size of convertion region: 0 (0)
Mon Nov 23 17:40:25 2015 [INFO]   Number of boot sectors backuped: 16 sectors (0x10)
Mon Nov 23 17:40:25 2015 [INFO]   First metadata header offset:  0x1f200000
Mon Nov 23 17:40:25 2015 [INFO]   Second metadata header offset: 0x5f200000
Mon Nov 23 17:40:25 2015 [INFO]   Third metadata header offset:  0x9f7fc000
Mon Nov 23 17:40:25 2015 [INFO]   Boot sectors backup address:   0x1f210000
Mon Nov 23 17:40:25 2015 [INFO]   ----------------------------{ Dataset header }----------------------------
Mon Nov 23 17:40:25 2015 [INFO]     Dataset size: 0x0000033a (826) bytes (including data)
Mon Nov 23 17:40:25 2015 [INFO]     Unknown data: 0x00000001 (always 0x00000001)
Mon Nov 23 17:40:25 2015 [INFO]     Dataset header size: 0x00000030 (always 0x00000030)
Mon Nov 23 17:40:25 2015 [INFO]     Dataset copy size: 0x0000033a (826) bytes
Mon Nov 23 17:40:25 2015 [INFO]     Dataset GUID: [...]
Mon Nov 23 17:40:25 2015 [INFO]     Next counter: 10
Mon Nov 23 17:40:25 2015 [INFO]     Encryption Type: AES-128-NODIFFUSER (0x8002)
Mon Nov 23 17:40:25 2015 [INFO]     Epoch Timestamp: 1438841786 sec, that to say Thu Aug  6 06:16:26 2015
Mon Nov 23 17:40:25 2015 [INFO]   --------------------------------------------------------------------------
Mon Nov 23 17:40:25 2015 [INFO] =============================================================================
Mon Nov 23 17:40:25 2015 [INFO] 
Mon Nov 23 17:40:25 2015 [INFO] 
Mon Nov 23 17:40:25 2015 [INFO] =======[ Datum n°1 informations ]=======
[...]

Do you think it might have run into a BitLocker metadata header? I see those are also 8192 bytes.

The corruption is ~568,303,648 bytes into a 605,594,334 byte file.

@ivan

This comment has been minimized.

Show comment
Hide comment
@ivan

ivan Nov 23, 2015

Probably not a surprise BitLocker metadata header. I checked and I have headers at the three metadata header offsets listed, but not near the 2905324924928 and 2778237616128 positions listed in the strace. (Though I wonder why ntfs-3g is reading from two completely different positions for an unfragmented file?)

ivan commented Nov 23, 2015

Probably not a surprise BitLocker metadata header. I checked and I have headers at the three metadata header offsets listed, but not near the 2905324924928 and 2778237616128 positions listed in the strace. (Though I wonder why ntfs-3g is reading from two completely different positions for an unfragmented file?)

@ivan

This comment has been minimized.

Show comment
Hide comment
@ivan

ivan Nov 23, 2015

(Though I wonder why ntfs-3g is reading from two completely different positions for an unfragmented file?)

Oops, probably just looking at calls for metadata instead of data.

ivan commented Nov 23, 2015

(Though I wonder why ntfs-3g is reading from two completely different positions for an unfragmented file?)

Oops, probably just looking at calls for metadata instead of data.

@ivan

This comment has been minimized.

Show comment
Hide comment
@ivan

ivan Nov 23, 2015

On a fresh mount.ntfs-3g process that hasn't cached any data yet, doing a read of the garbled 8KB section shows this strace:

read(4, "8\0\0\0\3\0\0\0\16\0\0\0\0\0\0\0\3\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 56
writev(4, [{"x\0\0\0\0\0\0\0\16\0\0\0\0\0\0\0", 16}, {"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\232\372;\0\0\0\0\0\0\0\2\0\0\0\0\0"..., 104}], 2) = 120
read(4, "\207\0\0\0\1\0\0\0\17\0\0\0\0\0\0\0\3\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 135
pread(3, "INDX(\0\t\0h\0019\366\10\0\0\0\3\0\0\0\0\0\0\0(\0\0\0\0\10\0\0"..., 4096, 2905324924928) = 4096
pread(3, "INDX(\0\t\0)\3054\366\10\0\0\0\t\0\0\0\0\0\0\0(\0\0\0\340\7\0\0"..., 4096, 2778115530752) = 4096
pread(3, "FILE0\0\3\0Y\3275\226\t\0\0\0\1\0\2\0008\0\1\0\200\2\0\0\0\4\0\0"..., 1024, 2830561321984) = 1024
writev(4, [{"\220\0\0\0\0\0\0\0\17\0\0\0\0\0\0\0", 16}, {"\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 128}], 2) = 144
read(4, "8\0\0\0\3\0\0\0\20\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 56
writev(4, [{"x\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0", 16}, {"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0Y\373;\0\0\0\0\0\336\242\30$\0\0\0\0"..., 104}], 2) = 120
read(4, "0\0\0\0\16\0\0\0\21\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 48
writev(4, [{" \0\0\0\0\0\0\0\21\0\0\0\0\0\0\0", 16}, {"\0\0\0\0\0\0\0\0\2\0\0\0\0\0\0\0", 16}], 2) = 32
read(4, "8\0\0\0\3\0\0\0\22\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 56
writev(4, [{"x\0\0\0\0\0\0\0\22\0\0\0\0\0\0\0", 16}, {"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0Y\373;\0\0\0\0\0\336\242\30$\0\0\0\0"..., 104}], 2) = 120
read(4, "8\0\0\0\3\0\0\0\23\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 56
writev(4, [{"x\0\0\0\0\0\0\0\23\0\0\0\0\0\0\0", 16}, {"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0Y\373;\0\0\0\0\0\336\242\30$\0\0\0\0"..., 104}], 2) = 120
read(4, "P\0\0\0\17\0\0\0\24\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 80
pread(3, "\360]P\345\216D?#>..\nO4\335P\264\346\3263X\314XI\326\333\245\230\356E\340r"..., 4096, 2199023247360) = 4096
writev(4, [{"\20\20\0\0\0\0\0\0\24\0\0\0\0\0\0\0", 16}, {"\360]P\345\216D?#>..\nO4\335P\264\346\3263X\314XI\326\333\245\230\356E\340r"..., 4096}], 2) = 4112
read(4, "P\0\0\0\17\0\0\0\25\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 80
pread(3, "?\24\324\1@\332\30p\10vct\177w\265\31\177\213\t:\236\25\316^\256`\23\276-;\322\241"..., 32768, 2199023251456) = 32768
writev(4, [{"\20\200\0\0\0\0\0\0\25\0\0\0\0\0\0\0", 16}, {"?\24\324\1@\332\30p\10vct\177w\265\31\177\213\t:\236\25\316^\256`\23\276-;\322\241"..., 32768}], 2) = 32784
read(4, "@\0\0\0\31\0\0\0\26\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 64
writev(4, [{"\20\0\0\0\332\377\377\377\26\0\0\0\0\0\0\0", 16}], 1) = 16
read(4, "@\0\0\0\22\0\0\0\27\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 135168) = 64
writev(4, [{"\20\0\0\0\0\0\0\0\27\0\0\0\0\0\0\0", 16}], 1) = 16
read(4, "8\0\0\0\3\0\0\0\30\0\0\0\0\0\0\0\3\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 56
writev(4, [{"x\0\0\0\0\0\0\0\30\0\0\0\0\0\0\0", 16}, {"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\232\372;\0\0\0\0\0\0\0\2\0\0\0\0\0"..., 104}], 2) = 120
read(4, "-\0\0\0\1\0\0\0\31\0\0\0\0\0\0\0\3\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
pread(3, "INDX(\0\t\0h\0019\366\10\0\0\0\3\0\0\0\0\0\0\0(\0\0\0\0\10\0\0"..., 4096, 2905324924928) = 4096
pread(3, "INDX(\0\t\0\204\0051\366\10\0\0\0\0\0\0\0\0\0\0\0(\0\0\0\250\7\0\0"..., 4096, 2778237616128) = 4096
writev(4, [{"\20\0\0\0\376\377\377\377\31\0\0\0\0\0\0\0", 16}], 1) = 16

The data read is at 2199023247360, which is exactly at the 2TB boundary:

>>> (2199023247360+8192)/(1024*1024*1024.0)
2048.0

ivan commented Nov 23, 2015

On a fresh mount.ntfs-3g process that hasn't cached any data yet, doing a read of the garbled 8KB section shows this strace:

read(4, "8\0\0\0\3\0\0\0\16\0\0\0\0\0\0\0\3\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 56
writev(4, [{"x\0\0\0\0\0\0\0\16\0\0\0\0\0\0\0", 16}, {"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\232\372;\0\0\0\0\0\0\0\2\0\0\0\0\0"..., 104}], 2) = 120
read(4, "\207\0\0\0\1\0\0\0\17\0\0\0\0\0\0\0\3\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 135
pread(3, "INDX(\0\t\0h\0019\366\10\0\0\0\3\0\0\0\0\0\0\0(\0\0\0\0\10\0\0"..., 4096, 2905324924928) = 4096
pread(3, "INDX(\0\t\0)\3054\366\10\0\0\0\t\0\0\0\0\0\0\0(\0\0\0\340\7\0\0"..., 4096, 2778115530752) = 4096
pread(3, "FILE0\0\3\0Y\3275\226\t\0\0\0\1\0\2\0008\0\1\0\200\2\0\0\0\4\0\0"..., 1024, 2830561321984) = 1024
writev(4, [{"\220\0\0\0\0\0\0\0\17\0\0\0\0\0\0\0", 16}, {"\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 128}], 2) = 144
read(4, "8\0\0\0\3\0\0\0\20\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 56
writev(4, [{"x\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0", 16}, {"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0Y\373;\0\0\0\0\0\336\242\30$\0\0\0\0"..., 104}], 2) = 120
read(4, "0\0\0\0\16\0\0\0\21\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 48
writev(4, [{" \0\0\0\0\0\0\0\21\0\0\0\0\0\0\0", 16}, {"\0\0\0\0\0\0\0\0\2\0\0\0\0\0\0\0", 16}], 2) = 32
read(4, "8\0\0\0\3\0\0\0\22\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 56
writev(4, [{"x\0\0\0\0\0\0\0\22\0\0\0\0\0\0\0", 16}, {"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0Y\373;\0\0\0\0\0\336\242\30$\0\0\0\0"..., 104}], 2) = 120
read(4, "8\0\0\0\3\0\0\0\23\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 56
writev(4, [{"x\0\0\0\0\0\0\0\23\0\0\0\0\0\0\0", 16}, {"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0Y\373;\0\0\0\0\0\336\242\30$\0\0\0\0"..., 104}], 2) = 120
read(4, "P\0\0\0\17\0\0\0\24\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 80
pread(3, "\360]P\345\216D?#>..\nO4\335P\264\346\3263X\314XI\326\333\245\230\356E\340r"..., 4096, 2199023247360) = 4096
writev(4, [{"\20\20\0\0\0\0\0\0\24\0\0\0\0\0\0\0", 16}, {"\360]P\345\216D?#>..\nO4\335P\264\346\3263X\314XI\326\333\245\230\356E\340r"..., 4096}], 2) = 4112
read(4, "P\0\0\0\17\0\0\0\25\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 80
pread(3, "?\24\324\1@\332\30p\10vct\177w\265\31\177\213\t:\236\25\316^\256`\23\276-;\322\241"..., 32768, 2199023251456) = 32768
writev(4, [{"\20\200\0\0\0\0\0\0\25\0\0\0\0\0\0\0", 16}, {"?\24\324\1@\332\30p\10vct\177w\265\31\177\213\t:\236\25\316^\256`\23\276-;\322\241"..., 32768}], 2) = 32784
read(4, "@\0\0\0\31\0\0\0\26\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 64
writev(4, [{"\20\0\0\0\332\377\377\377\26\0\0\0\0\0\0\0", 16}], 1) = 16
read(4, "@\0\0\0\22\0\0\0\27\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 135168) = 64
writev(4, [{"\20\0\0\0\0\0\0\0\27\0\0\0\0\0\0\0", 16}], 1) = 16
read(4, "8\0\0\0\3\0\0\0\30\0\0\0\0\0\0\0\3\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 56
writev(4, [{"x\0\0\0\0\0\0\0\30\0\0\0\0\0\0\0", 16}, {"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\232\372;\0\0\0\0\0\0\0\2\0\0\0\0\0"..., 104}], 2) = 120
read(4, "-\0\0\0\1\0\0\0\31\0\0\0\0\0\0\0\3\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 45
pread(3, "INDX(\0\t\0h\0019\366\10\0\0\0\3\0\0\0\0\0\0\0(\0\0\0\0\10\0\0"..., 4096, 2905324924928) = 4096
pread(3, "INDX(\0\t\0\204\0051\366\10\0\0\0\0\0\0\0\0\0\0\0(\0\0\0\250\7\0\0"..., 4096, 2778237616128) = 4096
writev(4, [{"\20\0\0\0\376\377\377\377\31\0\0\0\0\0\0\0", 16}], 1) = 16

The data read is at 2199023247360, which is exactly at the 2TB boundary:

>>> (2199023247360+8192)/(1024*1024*1024.0)
2048.0

@ivan ivan changed the title from Two 4KB blocks garbled in one file to 8KB of data garbled at 2TB boundary Nov 23, 2015

@ivan

This comment has been minimized.

Show comment
Hide comment
@ivan

ivan Nov 23, 2015

Running dislocker with DEBUG logging shows this when reading the troubled 8KB section:

Mon Nov 23 19:51:04 2015 [DEBUG] --------------------{ Fuse reading }-----------------------
Mon Nov 23 19:51:04 2015 [DEBUG]   Offset and size needed: 0x286d496e000 and 0x1000
Mon Nov 23 19:51:04 2015 [DEBUG]   Start sector number: 0x1436a4b70 || Number of sectors: 0x8
Mon Nov 23 19:51:04 2015 [DEBUG]   Trying to allocate 0x1000 bytes
Mon Nov 23 19:51:04 2015 [DEBUG]   Outsize which will be returned: 4096
Mon Nov 23 19:51:04 2015 [DEBUG] -----------------------------------------------------------
Mon Nov 23 19:51:04 2015 [DEBUG] --------------------{ Fuse reading }-----------------------
Mon Nov 23 19:51:04 2015 [DEBUG]   Offset and size needed: 0x2930a9a2000 and 0x1000
Mon Nov 23 19:51:04 2015 [DEBUG]   Start sector number: 0x149854d10 || Number of sectors: 0x8
Mon Nov 23 19:51:04 2015 [DEBUG]   Trying to allocate 0x1000 bytes
Mon Nov 23 19:51:04 2015 [DEBUG]   Outsize which will be returned: 4096
Mon Nov 23 19:51:04 2015 [DEBUG] -----------------------------------------------------------
Mon Nov 23 19:51:04 2015 [DEBUG] --------------------{ Fuse reading }-----------------------
Mon Nov 23 19:51:04 2015 [DEBUG]   Offset and size needed: 0x20000000000 and 0x1000
Mon Nov 23 19:51:04 2015 [DEBUG]   Start sector number: 0x100000000 || Number of sectors: 0x8
Mon Nov 23 19:51:04 2015 [DEBUG]   Trying to allocate 0x1000 bytes
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000000000 to 0x2001f210000
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000000200 to 0x2001f210200
Mon Nov 23 19:51:04 2015 [DEBUG] Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000000400 to 0x2001f210400
  Fixing sector (7): from 0x20000000600 to 0x2001f210600
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000000800 to 0x2001f210800
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000000a00 to 0x2001f210a00
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000000c00 to 0x2001f210c00
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000000e00 to 0x2001f210e00
Mon Nov 23 19:51:04 2015 [DEBUG]   Outsize which will be returned: 4096
Mon Nov 23 19:51:04 2015 [DEBUG] -----------------------------------------------------------
Mon Nov 23 19:51:04 2015 [DEBUG] --------------------{ Fuse reading }-----------------------
Mon Nov 23 19:51:04 2015 [DEBUG]   Offset and size needed: 0x20000001000 and 0x8000
Mon Nov 23 19:51:04 2015 [DEBUG]   Start sector number: 0x100000008 || Number of sectors: 0x40
Mon Nov 23 19:51:04 2015 [DEBUG]   Trying to allocate 0x8000 bytes
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000001000 to 0x2001f211000
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000001200 to 0x2001f211200
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000001400 to 0x2001f211400
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000001600 to 0x2001f211600
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000001800 to 0x2001f211800
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000001a00 to 0x2001f211a00
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000001c00 to 0x2001f211c00
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000001e00 to 0x2001f211e00
Mon Nov 23 19:51:04 2015 [DEBUG]   Outsize which will be returned: 32768
Mon Nov 23 19:51:04 2015 [DEBUG] -----------------------------------------------------------

It looks like it's calling into fix_read_sector_seven when reading at the 2TB boundary. I see there's a (uint32_t)sector_offset conversion in src/inouts/sectors.c.

ivan commented Nov 23, 2015

Running dislocker with DEBUG logging shows this when reading the troubled 8KB section:

Mon Nov 23 19:51:04 2015 [DEBUG] --------------------{ Fuse reading }-----------------------
Mon Nov 23 19:51:04 2015 [DEBUG]   Offset and size needed: 0x286d496e000 and 0x1000
Mon Nov 23 19:51:04 2015 [DEBUG]   Start sector number: 0x1436a4b70 || Number of sectors: 0x8
Mon Nov 23 19:51:04 2015 [DEBUG]   Trying to allocate 0x1000 bytes
Mon Nov 23 19:51:04 2015 [DEBUG]   Outsize which will be returned: 4096
Mon Nov 23 19:51:04 2015 [DEBUG] -----------------------------------------------------------
Mon Nov 23 19:51:04 2015 [DEBUG] --------------------{ Fuse reading }-----------------------
Mon Nov 23 19:51:04 2015 [DEBUG]   Offset and size needed: 0x2930a9a2000 and 0x1000
Mon Nov 23 19:51:04 2015 [DEBUG]   Start sector number: 0x149854d10 || Number of sectors: 0x8
Mon Nov 23 19:51:04 2015 [DEBUG]   Trying to allocate 0x1000 bytes
Mon Nov 23 19:51:04 2015 [DEBUG]   Outsize which will be returned: 4096
Mon Nov 23 19:51:04 2015 [DEBUG] -----------------------------------------------------------
Mon Nov 23 19:51:04 2015 [DEBUG] --------------------{ Fuse reading }-----------------------
Mon Nov 23 19:51:04 2015 [DEBUG]   Offset and size needed: 0x20000000000 and 0x1000
Mon Nov 23 19:51:04 2015 [DEBUG]   Start sector number: 0x100000000 || Number of sectors: 0x8
Mon Nov 23 19:51:04 2015 [DEBUG]   Trying to allocate 0x1000 bytes
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000000000 to 0x2001f210000
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000000200 to 0x2001f210200
Mon Nov 23 19:51:04 2015 [DEBUG] Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000000400 to 0x2001f210400
  Fixing sector (7): from 0x20000000600 to 0x2001f210600
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000000800 to 0x2001f210800
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000000a00 to 0x2001f210a00
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000000c00 to 0x2001f210c00
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000000e00 to 0x2001f210e00
Mon Nov 23 19:51:04 2015 [DEBUG]   Outsize which will be returned: 4096
Mon Nov 23 19:51:04 2015 [DEBUG] -----------------------------------------------------------
Mon Nov 23 19:51:04 2015 [DEBUG] --------------------{ Fuse reading }-----------------------
Mon Nov 23 19:51:04 2015 [DEBUG]   Offset and size needed: 0x20000001000 and 0x8000
Mon Nov 23 19:51:04 2015 [DEBUG]   Start sector number: 0x100000008 || Number of sectors: 0x40
Mon Nov 23 19:51:04 2015 [DEBUG]   Trying to allocate 0x8000 bytes
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000001000 to 0x2001f211000
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000001200 to 0x2001f211200
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000001400 to 0x2001f211400
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000001600 to 0x2001f211600
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000001800 to 0x2001f211800
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000001a00 to 0x2001f211a00
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000001c00 to 0x2001f211c00
Mon Nov 23 19:51:04 2015 [DEBUG]   Fixing sector (7): from 0x20000001e00 to 0x2001f211e00
Mon Nov 23 19:51:04 2015 [DEBUG]   Outsize which will be returned: 32768
Mon Nov 23 19:51:04 2015 [DEBUG] -----------------------------------------------------------

It looks like it's calling into fix_read_sector_seven when reading at the 2TB boundary. I see there's a (uint32_t)sector_offset conversion in src/inouts/sectors.c.

@ivan

This comment has been minimized.

Show comment
Hide comment
@ivan

ivan Nov 23, 2015

I added some more debug lines, and I see

offset=0x20000000000
sector_offset=0x100000000
(uint32_t)sector_offset=0x0
sector_size=512

ivan commented Nov 23, 2015

I added some more debug lines, and I see

offset=0x20000000000
sector_offset=0x100000000
(uint32_t)sector_offset=0x0
sector_size=512

@ivan

This comment has been minimized.

Show comment
Hide comment
@ivan

ivan Nov 23, 2015

My file is read correctly after changing src/inouts/sectors.c:

        if(version == V_SEVEN &&
-          (uint32_t)sector_offset < io_data->nb_backup_sectors)
+          (uint64_t)sector_offset < io_data->nb_backup_sectors)
        {

ivan commented Nov 23, 2015

My file is read correctly after changing src/inouts/sectors.c:

        if(version == V_SEVEN &&
-          (uint32_t)sector_offset < io_data->nb_backup_sectors)
+          (uint64_t)sector_offset < io_data->nb_backup_sectors)
        {
@ivan

This comment has been minimized.

Show comment
Hide comment
@ivan

ivan Nov 23, 2015

Regressed in d9174c8

ivan commented Nov 23, 2015

Regressed in d9174c8

@Aorimn

This comment has been minimized.

Show comment
Hide comment
@Aorimn

Aorimn Nov 24, 2015

Owner

Wowkay, nice catch! I'll have to make some tests to revert this commit but I'll definitely fix that (thanks to you!).

Owner

Aorimn commented Nov 24, 2015

Wowkay, nice catch! I'll have to make some tests to revert this commit but I'll definitely fix that (thanks to you!).

@Aorimn Aorimn added the bug label Nov 25, 2015

Aorimn added a commit that referenced this issue Nov 25, 2015

Revert "Fix integer conversion"
This reverts commit d9174c8 due to
reported issue #51.
@Aorimn

This comment has been minimized.

Show comment
Hide comment
@Aorimn

Aorimn Nov 25, 2015

Owner

@ivan I just pushed your suggested patch by reverting d9174c8. From the tests I've done, it's okay, so I'm not sure why I did this commit - maybe a compatibility issue with Ubuntu 10.04 LTS at the time.

Owner

Aorimn commented Nov 25, 2015

@ivan I just pushed your suggested patch by reverting d9174c8. From the tests I've done, it's okay, so I'm not sure why I did this commit - maybe a compatibility issue with Ubuntu 10.04 LTS at the time.

@Aorimn

This comment has been minimized.

Show comment
Hide comment
@Aorimn

Aorimn Nov 25, 2015

Owner

Thanks for your report!

Owner

Aorimn commented Nov 25, 2015

Thanks for your report!

@Aorimn Aorimn closed this Nov 25, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment