New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fsremap crashes on large filesystems #3

Closed
ccantill opened this Issue Nov 21, 2014 · 3 comments

Comments

Projects
None yet
2 participants
@ccantill

ccantill commented Nov 21, 2014

I tried converting my 9.1TB JFS filesystem to EXT4 (so I could shrink it) and after three days of churning it finally got to the fsremap part which promptly crashed:

17:55:22 fstransform: mounting again device '/dev/md0' read-only
17:55:22 fstransform: launching '/usr/sbin/fsremap' in simulated mode
17:55:22 fsremap: starting job 1, persistence data and logs are in '/var/tmp/fstransform/fsremap.job.1'
17:55:22 fsremap: if this job is interrupted, for example by a power failure,
17:55:22 fsremap: you CAN RESUME it with: /usr/sbin/fsremap -n -q --resume-job=1 -- /dev/md0
17:55:23 fsremap: ERROR: ff_posix_fibmap(): error, dev_block_count = 7, file_block_count = 2441889920 overflow type (int): File too large
17:55:23 fsremap: ERROR: failed to list file blocks with ioctl(FS_IOC_FIEMAP): Operation not supported

@ccantill

This comment has been minimized.

ccantill commented Nov 21, 2014

I did some more digging. My JFS volume is 19534457248 blocks in size, so too large to fit in an int (or even an unsigned int for that matter). So using FIBMAP is out of the question since there still doesn't appear to be a 64bit alternative for it. However (but I'm not expert in this area at all) I think FIEMAP might be a better fit here anyway; it uses 64-bit unsigned integers and returns whole extents instead of blocks so it could be faster too.

@cosmos72

This comment has been minimized.

Owner

cosmos72 commented Nov 22, 2014

Hello ccantill,

yes, fsremap tries FIEMAP first. As you can see from the last error line you quoted, current JFS kernel driver on Linux does not support FIEMAP.
Lacking FIEMAP, fsremap then falls back on FIBMAP, which is limited by design to 2^31-1 blocks. In your case blocks are 4k, so this translates to a 8TB limit, not enough for your 9.1TB partition.

Without usable FIEMAP nor FIBMAP, fsremap cannot proceed and aborts.

Your data is still accessible, even if a bit "hidden": it's stored in an EXT4 file ".fstransform.loop.[SOME-NUMBER]" inside the JFS file system: you can simply loop-mount it and access its contents. If your JFS file system is less than 50% full, you can also copy the data back; otherwise you will need additional storage to copy it.

I hope you understand why I placed a BIG DISCLAIMER on fstransform: while I tried to be as thorough as possible with tests and error checks, there's no way I can guarantee that they cover 100% of the cases. And people keep surprising me with "extreme" cases... Anyway this case is easy to detect, I will add an early check for it.

@ccantill

This comment has been minimized.

ccantill commented Nov 22, 2014

Ah, I missed that bit about FIEMAP. I wasn't worried about losing data. I'm converting my RAID5 to a JBOD setup with snapshot RAID so I wanted to shrink my RAID5 and move data to the individual volumes but JFS doesn't shrink so I thought this would help. Guess I'll have to find some more spare storage to move everything to as a go-between. Thanks for the great tool anyway :)

@cosmos72 cosmos72 closed this Nov 22, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment