New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NTFS > EXT4 target busy issue #18

Open
asanghc opened this Issue Apr 22, 2018 · 11 comments

Comments

Projects
None yet
2 participants
@asanghc

asanghc commented Apr 22, 2018

Ubuntu 17.10 on a remote 10 y/o asus laptop
fstransform 0.9.3
/dev/SDB is and 1TB external USB 3.0 spinning HDD

I launched these two commands in two tmux terminals about a week ago:
fstransform --old-file-system=ntfs --force-untested-file-systems /dev/sdb1 ext4
(the command proposed to monitor the loop file size - it never exceeded 60%)
the conversion went to 100% and looked like it would succeed yesterday but then it threw these errors:
02:55:14 fstransform: disconnected loop device '/dev/loop1' from file '/tmp/fstransform.mount.7857/.fstransform.loop.8107'
02:55:14 fstransform: unmounting device '/dev/sdb1' before disk check
02:55:14 umount: /tmp/fstransform.mount.7857: target is busy.

02:55:14 ERROR! fstransform: command '/bin/umount /dev/sdb1' failed (exit status 32)
this is potentially a problem.
you can either quit now by pressing ENTER or CTRL+C,

            or, if you know what went wrong, you can fix it yourself,
            then manually run the command '/bin/umount /dev/sdb1'
            (or something equivalent)
            and finally resume this script by typing CONTINUE and pressing ENTER: CONTINUE

12:47:23 fstransform: running '/sbin/fsck' (disk check) on device '/dev/sdb1'
12:47:23 fsck: fsck from util-linux 2.31.1
12:47:23 fstransform: mounting again device '/dev/sdb1' read-only
12:47:24 mount: ntfs-3g-mount: mount failed: Device or resource busy

12:47:24 ERROR! fstransform: command '/bin/mount /dev/sdb1 /tmp/fstransform.mount.7857 -o ro' failed (exit status 21)
this is potentially a problem.
you can either quit now by pressing ENTER or CTRL+C,

            or, if you know what went wrong, you can fix it yourself,
            then manually run the command '/bin/mount /dev/sdb1 /tmp/fstransform.mount.7857 -o ro'
            (or something equivalent)
            and finally resume this script by typing CONTINUE and pressing ENTER: CONTINUE

16:22:09 fstransform: launching '/usr/sbin/fsremap' in simulated mode
16:22:09 fsremap: starting job 1, persistence data and logs are in '/var/tmp/fstransform/fsremap.job.1'
16:22:09 fsremap: if this job is interrupted, for example by a power failure,
16:22:09 fsremap: you CAN RESUME it with: /usr/sbin/fsremap -n -q --resume-job=1 -- /dev/sdb1
16:22:09 fsremap: ERROR: error opening loop-file '/tmp/fstransform.mount.7857/.fstransform.loop.8107': No such file or directory

16:22:09 ERROR! fstransform: command '/usr/sbin/fsremap -n -q -- /dev/sdb1 /tmp/fstransform.mount.7857/.fstransform.loop.8107' failed (exit status 254)
this is potentially a problem.
you can either quit now by pressing ENTER or CTRL+C,

            or, if you know what went wrong, you can fix it yourself,
            then manually run the command '/usr/sbin/fsremap -n -q -- /dev/sdb1 /tmp/fstransform.mount.7857/.fstransform.loop.8107'
            (or something equivalent)
            and finally resume this script by typing CONTINUE and pressing ENTER: CONTINUE

16:39:36 fstransform: launching '/usr/sbin/fsremap' in REAL mode to perform in-place remapping.
16:39:36 fsremap: starting job 2, persistence data and logs are in '/var/tmp/fstransform/fsremap.job.2'
16:39:36 fsremap: if this job is interrupted, for example by a power failure,
16:39:36 fsremap: you CAN RESUME it with: /usr/sbin/fsremap -q --resume-job=2 -- /dev/sdb1
16:39:36 fsremap: ERROR: error opening loop-file '/tmp/fstransform.mount.7857/.fstransform.loop.8107': Transport endpoint is not connected

16:39:36 ERROR! fstransform: command '/usr/sbin/fsremap -q --cmd-umount=/bin/umount -- /dev/sdb1 /tmp/fstransform.mount.7857/.fstransform.loop.8107' failed (exit status 149)
this is potentially a problem.
you can either quit now by pressing ENTER or CTRL+C,

            or, if you know what went wrong, you can fix it yourself,
            then manually run the command '/usr/sbin/fsremap -q --cmd-umount=/bin/umount -- /dev/sdb1 /tmp/fstransform.mount.7857/.fstransform.loop.8107'
            (or something equivalent)
            and finally resume this script by typing CONTINUE and pressing ENTER:"

I'll admit I'm lost but these are the steps I attempted to fix it:
-killed anything I thought could be keeping /dev/sdb1 busy
-after running out of ideas, forced the umount with -l (lazy)
-problem still wasn't fixed with that
-I ended up force remounting the drive with -o force (probably a stupid idea)
-commented out entries in fstab that mount that drive
-rebooted (another stupid idea probably)
-now any attempts to sudo fsremap -q --resume-job=1|2|3 /dev/sdb1 fail with
ERROR: error opening loop-file '(null)': Bad address
I'm asking for help here now to try to avoid making things worse (maybe too late?)

What happened and can you halp?!

@cosmos72

This comment has been minimized.

Owner

cosmos72 commented Apr 22, 2018

Hello @asanghc,

You chose a very unlucky combination:

  • NTFS support is experimental and untested - which means dangerous - in fact you added the option --force-untested-file-systems because by default fstransform would refuse to convert it.

  • There is a known issue with fstransform on Ubuntu: by default, almost all versions of Ubuntu open a file manager window on each newly available partition, including loop files. This prevents fstransform from unmounting the partition to perform the fsremap - exactly the error you reported. And since fsremap is never started properly, --resume-job does not find anything to resume.

Despite these issues, and depending on the contents of your /dev/sdb1, it may still be possible to complete the conversion. Here are the steps:

  1. if you have a Windows installation available, perform a disk check on /dev/sdb1 from Windows.
  2. on Ubuntu, execute as root the commands: mkdir /mnt; mount /dev/sdb1 /mnt; ls -al /mnt
  3. if the output of ls command above does not show any file, you are in trouble.
  4. if instead it shows the files and directories that were initially on the disk before the conversion, but no file name .fstransform.loop.* (unlikely), then you still have an NTFS volume with your data.
  5. if instead there is exactly one file, hopefully named .fstransform.loop.8107, keep reading:
  6. perform a filesystem check on the file /mnt/.fstransform.loop.8107 - or whatever is its name: e2fsck -C0 -f /mnt/.fstransform.loop.8107
  7. if the filesystem is clean or gets successfully repaired, keep reading. Otherwise you are in trouble.
  8. close any file manager window that Ubuntu opened.
  9. execute umount /mnt
  10. if you have a recent fstransform that includes the command fsmount_kernel, execute the command fsmount_kernel -o ro -t ntfs /dev/sdb1 /mnt. Otherwise execute the "regular" command mount -o ro /dev/sdb1 /mnt. Note: fsmount_kernel is strongly preferred because fsremap will be faster with it.
  11. again, close any file manager window that Ubuntu opened.
  12. run the command that was printed at the initial error, updated for the fact that /dev/sdb1 is now mounted at /mnt. the command is: fsremap --cmd-umount=/bin/umount -- /dev/sdb1 /mnt/.fstransform.loop.8107 (or whatever is the name of the file you found at step 6)
  13. be patient. The conversion of 1TB will likely need 20-30 hours.
  14. if you get an error similar to umount: /mnt: target is busy, do not type anything in the terminal where fsremap is running. Instead, locate all file manager windows that Ubuntu opened and close them. Then, in a different terminal, try again umount /mnt and, if it still fails, run fuser -vm /mnt to find which processes are using /mnt and kill them. Once you finally unmounted /mnt, go back to the terminal where fremap is running, and type uppercase CONTINUE then press enter. Conversion should resume.
@cosmos72

This comment has been minimized.

Owner

cosmos72 commented Apr 22, 2018

Let me know the outcome :)

@asanghc

This comment has been minimized.

asanghc commented Apr 24, 2018

Thank you for the quick reply. I'll try your advise this weekend. (never enough time) This is a remote system which I can only access via SSH. This laptop has no monitor (destroyed in accident) and runs only ubuntu server with no X or GUI. I'm not sure what could have cause the "target busy" issue. I'm not suggesting that your awesome tool might be at fault, I just don't know exactly how I screwed this up. Cheers

@asanghc

This comment has been minimized.

asanghc commented Apr 29, 2018

Hi again, since #5 is true, I did #6 successfully and since I have version 0.9.3, it doesn't look like I can use the quicker fsmount_kernel mode. Am I making sense so far? Above which versions is fsmount_kernel available? Must I compile it or is there a newer version available in .deb form or through a ppa? Thank you very much for your assistance.

@cosmos72

This comment has been minimized.

Owner

cosmos72 commented Apr 29, 2018

Good news :)
Yes, it makes sense.

fsmount_kernel is "just" a speedup. There is no updated .deb package containing it, you would have to compile it from sources: you can download the latest fstransform from https://github.com/cosmos72/fstransform/archive/master.tar.gz, uncompress (tar -xzf master.tar.gz), configure (cd fstransform; ./configure), compile (make) and install it (sudo make install).
If you are not comfortable with compiling programs from source, you can do without it and follow the procedure described above anyway.

@asanghc

This comment has been minimized.

asanghc commented Jul 23, 2018

I compiled/installed per your instructions without problems or errors.
everything goes well for steps 5 through 12 but not proceeding beyond that.

sudo fsremap --cmd-umount=/bin/umount -- /dev/sdb1 /mnt/.fstransform.loop.8107
setting log level to INFO
fsremap: starting job 6, persistence data and logs are in '/var/tmp/fstransform/fsremap.job.6'
if this job is interrupted, for example by a power failure,
you CAN RESUME it with: fsremap -q --resume-job=6 -- /dev/sdb1
device length is 931.48 gigabytes
analyzing device '/dev/sdb1', this may take some minutes ...

it remains stuck at this step indefinitely

@cosmos72

This comment has been minimized.

Owner

cosmos72 commented Jul 23, 2018

I guess you executed fsremap with /dev/sdb1 already mounted, right?

In such case, whatever kernel driver is in use for NTFS will be employed for the analysis phase - usually, FUSE ntfs-3g, which as I wrote above, is extremely slow for the analysis.

The faster option I mentioned is (after killing the running fsremap stuck in the analysis phase), to execute as root:

umount /dev/sdb1
fsmount_kernel -o ro -t ntfs /dev/sdb1 /mnt
fsremap --cmd-umount=/bin/umount -- /dev/sdb1 /mnt/.fstransform.loop.8107

The umount command executed by fsremap after the analysis may fail if Ubuntu shows a file manager opened on /mnt (which it usually does) - in such case, close all file managers, umount /mnt manually, then type CONTINUE at fsremap prompt

@asanghc

This comment has been minimized.

asanghc commented Jul 23, 2018

Is there a way to monitor progress or at least get confirmation that SOMETHING is happening? This is using the faster fsmount_kernel option. I've tried both mount options and different versions of the fstransform tool and waited over a week each time.

This is a headless system without a GUI and I'm doing everything via SSH and tmux. The original "read only" issue is no longer an issue and there is never a prompt to CONTINUE

Would you be interested in seeing the contents of any of these files?
/var/tmp/fstransform$ ls
total 72K
4.0K drwxr-xr-x 9 root root 4.0K Jul 23 15:25 .
4.0K drwxrwxrwt 5 root root 4.0K Jul 22 07:49 ..
4.0K drwxr-xr-x 2 root root 4.0K Apr 22 16:22 fsremap.job.1
4.0K drwxr-xr-x 2 root root 4.0K Apr 22 16:39 fsremap.job.2
4.0K drwxr-xr-x 2 root root 4.0K Apr 22 16:40 fsremap.job.3
4.0K drwxr-xr-x 2 root root 4.0K Apr 29 22:21 fsremap.job.4
4.0K drwxr-xr-x 2 root root 4.0K Apr 30 18:32 fsremap.job.5
4.0K drwxr-xr-x 2 root root 4.0K Jul 23 12:12 fsremap.job.6
4.0K drwxr-xr-x 2 root root 4.0K Jul 23 15:25 fsremap.job.7
4.0K -rw-r--r-- 1 root root 1022 Apr 18 11:06 fstransform.log.7761
4.0K -rw-r--r-- 1 root root 3.4K Apr 18 11:08 fstransform.log.7857
28K -rw-r--r-- 1 root root 23K Apr 22 16:39 fstransform.log.8107

@asanghc

This comment has been minimized.

asanghc commented Jul 23, 2018

Does this explain anything?

2018-07-23 15:31:16 DEBUG [io/extent_posix.ff_linux_fiemap(189)] ioctl(6, FIEMAP, extents[1024]) failed (Operation not supported), falling back on ioctl(FIBMAP) ...

@cosmos72

This comment has been minimized.

Owner

cosmos72 commented Jul 23, 2018

You already listed the most relevant line, no need to send me the logs.

fsremap on NTFS takes ages to analyze the device because ioctl(FS_IOC_FIEMAP) is not supported, and ioctl(FIBMAP) is terribly slow on FUSE ntfs-3g. Using fsmount_kernel speeds up a little ioctl(FIBMAP), as the kernel implementation is merely slow instead of terribly slow.

To monitor the progress of this analysis phase, you can try attaching strace -v -T to the running fsremap and post here a small extract (a few lines showing some calls to ioctl(FIBMAP) are enough)

I understand your frustration: you are experiencing exactly the issues I met too, and that convinced me not to support NTFS (at least not yet). This also the reason why fstransform refuses to convert NTFS file systems unless you force it with --force-untested-file-systems

@cosmos72

This comment has been minimized.

Owner

cosmos72 commented Jul 23, 2018

Update: in case I did not mention it yet, since you are stuck at step #12, another option is to have a 1TB free partition on the same system (say /dev/sdx1) and perform as root direct copy of the huge loop file:

dd bs=10M if=/mnt/.fstransform.loop.8107 of=/dev/sdx1

If you have enough free space, this is the fastest solution

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment