Implement handling of non-inline file extents

[cblichmann]

Currently, only inline file data can be recovered, which is usually very small (< 4KiB). This obviously greatly limits the tool's usefulness.
Implement full support for file extents, block groups and chunks. For the simple, non-RAID case, this should be a matter of implementing the physicalToLogical() and logicalToPhysical() functions.

[vk496]

Would really love to see this implemented!

Do you have some scratch code to share and start from? Or some notes where to start? I would like to help making this basic functionally possible

[ghost]

I too want it.

[cblichmann]

Check out 1bc51ce. That commit allows to read the first file extent at least.

[vk496]

Hello,

Thnks @cblichmann. I have a HDD with a corrupted btrfs FS. Trying your last commit, I receive this error:

$ sudo ./btrfscue --metadata /srv/USB/btrfscue/metadata_4t.db --verbose  mount /dev/mapper/HDD test3/
mounted rescue FS on test3/
2019/02/18 15:37:25 Unknown opcode 2016
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x103adc]

goroutine 4 [running]:
blichmann.eu/code/btrfscue/vendor/github.com/hanwen/go-fuse/fuse.(*request).serializeHeader(0x40000e6000, 0x0, 0x28, 0x0, 0x0)
        /home/rock64/test2/src/blichmann.eu/code/btrfscue/vendor/github.com/hanwen/go-fuse/fuse/request.go:193 +0x1c
blichmann.eu/code/btrfscue/vendor/github.com/hanwen/go-fuse/fuse.(*Server).write(0x40000e00e0, 0x40000e6000, 0x4000000000)
        /home/rock64/test2/src/blichmann.eu/code/btrfscue/vendor/github.com/hanwen/go-fuse/fuse/server.go:428 +0x54
blichmann.eu/code/btrfscue/vendor/github.com/hanwen/go-fuse/fuse.(*Server).handleRequest(0x40000e00e0, 0x40000e6000, 0x40000e6000)
        /home/rock64/test2/src/blichmann.eu/code/btrfscue/vendor/github.com/hanwen/go-fuse/fuse/server.go:401 +0x6c
blichmann.eu/code/btrfscue/vendor/github.com/hanwen/go-fuse/fuse.(*Server).loop(0x40000e00e0, 0x400000c001)
        /home/rock64/test2/src/blichmann.eu/code/btrfscue/vendor/github.com/hanwen/go-fuse/fuse/server.go:377 +0x11c
created by blichmann.eu/code/btrfscue/vendor/github.com/hanwen/go-fuse/fuse.(*Server).readRequest
        /home/rock64/test2/src/blichmann.eu/code/btrfscue/vendor/github.com/hanwen/go-fuse/fuse/server.go:285 +0x230

when trying to ls inside "rescue" path

 sudo ls test3 
metadata  rescue
rock64@rock64:~/test2/bin$ sudo ls test3/rescue/
ls: cannot open directory 'test3/rescue/': Transport endpoint is not connected

Any idea?

EDIT:

I've checked with the previous commit and have same issue. Maybe I should open a new issue?

[cblichmann]

Any chance you can share that metadata_4t.db? If not (perfectly fine as well), let me do some digging myself. If you can/want to share, use my GPG key CF13896F.
I also should maybe update the vendoered go-fuse dependency :-)

[vk496]

Any chance you can share that metadata_4t.db? If not (perfectly fine as well), let me do some digging myself. If you can/want to share, use my GPG key CF13896F.
I also should maybe update the vendoered go-fuse dependency :-)

Yes, sure! But your GPG key revoked yesterday ^^

Waiting for a new one :)

[cblichmann]

I updated the key yesterday: https://blichmann.eu/christian-blichmann-public.asc

[vk496]

I updated the key yesterday: https://blichmann.eu/christian-blichmann-public.asc

Oks, thnks. When I checked, it was not available.

I sent you by email the file. Hope it will help you.

Salu2

[cblichmann]

Hi again,
I got your metadata_4t.db file and successfully decrypted it. I can't, however, reproduce the crash.
Mount:

(redacted)@(redacted):~/devel/btrfscue-tmp/__github$ ./btrfscue --metadata metadata_4t.db --verbose mount ~/mnt                                                                                                 
btrfscue: no device file given, only inline file data will be visible                                                                                                                                                        
mounted rescue FS on /home/(redacted)/mnt                                                                                                                                                                                          

List root directory (I stripped out some info, but surely you can recognize the filenames):

(redacted)@(redacted):~/mnt/rescue$ ls -alhn
ls: cannot access 'randomSTUFF': No data available
ls: cannot access '(redacted)': No data available
ls: cannot access 'rock64_SD.gz': No data available
ls: cannot access 'manjaro': No data available
total 0
drwxr-xr-x  1 1000 1000  552 Jan 21 23:00 .
dr-xr-xr-x  1    0    0    0 Feb 21 00:18 ..
drwxrwSr-x+ 1 1000  100  944 Jan 22 11:44 PC_BACKUP
drwxr-xr-x  1 1000 1000  802 Jan  3 00:31 VoiceRecorder
drwxr-xr-x  1    0    0   56 Jan  6 23:25 _openwrt-HW556-squashfs-cfe_r33275_wlan_restore.bin.extracted
drwxrwsr-x+ 1 1000 1000   74 Apr 23  2018 backup
drwxr-xr-x  1 1000 1000  162 Oct 25 18:39 backup_1T
d?????????? ? ?    ?       ?            ? (redacted)
drwxr-xr-x  1  999  999   22 Jan 21 19:07 btrfs_recovery
drwxr-xr-x  1 1000 1000   40 Jan  9 14:39 docker_cfg
drwxr-xr-x  1 1000    0   10 Sep 18 18:30 ext2_saved
d?????????? ? ?    ?       ?            ? manjaro
...(redacted)...

Looking through those folders, there are quite a few files/directories with incomplete data. This may mean that my code is buggy/incomplete and cannot find all the bits and pieces (likely) or that the data is actually damaged (probable). How was this damaged? Physical HDD failure or something different, like "I overwrote parts of it to test btrfscue"? :)

Regarding the crash, what's your precise environment? Which OS, Golang version were you using?

[vk496]

Hello,

You right, from my PC work correctly!

The previous test was from my Rock64.

$ uname -a
Linux rock64 4.4.154-1122-rockchip-ayufan-g7859b9b904a9 #1 SMP Mon Oct 1 07:32:42 UTC 2018 aarch64 GNU/Linux
$ go version
go version go1.11.5 linux/arm64

The FS was corrupted because of a light cut. I tried to recover the FS with btrfs tools and I 99% sure it destroyed it more.

PS: When I was copying some files, I was receiving this message:

btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
btrfscue: read out of bounds: only first extent supported (up to 30932992 bytes)
...

And for some files, this:

cp: no se puede efectuar `stat' sobre 'backup/Camera: No hay datos disponibles
...

[cblichmann]

I'll be working on this next, but I expect to get to this only in a few weeks.
The trouble is that there are still a few issues translating logical into physical byte offsets. We need to look for BTRFS chunks in the FS that have the same or later generation number than the file extents.