Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drive not detected in openSUSE Tumbleweed #31

Closed
ghost opened this issue Oct 8, 2016 · 10 comments
Closed

Drive not detected in openSUSE Tumbleweed #31

ghost opened this issue Oct 8, 2016 · 10 comments
Labels

Comments

@ghost
Copy link

ghost commented Oct 8, 2016

GNOME Files doesn't see the resulting drive of format-udf.sh as mountable.

In comparison: running mkudffs --media-type=hd --vid=Morrow --lvid=Morrow /dev/sdc produces a drive I can mount and read and write to in Linux.

NAME="openSUSE Tumbleweed"

VERSION="20161003"

udftools version is 1.0.0b3-161.4

Here's the result of:
./format-udf.sh sdc Morrow

[+] Testing dependencies...
[+] Looking for drive listing tool...which: no diskutil in (/usr/local/bin:/usr/bin:/bin:/usr/games:/sbin)
 using /sbin/blockdev
[+] Looking for unmount tool...which: no diskutil in (/usr/local/bin:/usr/bin:/bin:/usr/games:/sbin)
 using /usr/bin/umount
[+] Looking for UDF tool...which: no newfs_udf in (/usr/local/bin:/usr/bin:/bin:/usr/games:/sbin)
 using /usr/bin/mkudffs
[+] Parsing options...
[+] Validating arguments...
[+] Gathering drive information...
BUP Fast HDD    
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw  1024   512   512          0   4000797859328   /dev/sdc
The above-listed device (and partitions, if any) will be completely erased.
Type 'yes' if this is what you intend:  yes
[+] Detecting total size...
[*] Using total size of 4000797859328
[+] Validating detected total size...
[+] Detecting physical block size...
[*] Using block size of 4096
[+] Validating detected block size...
[+] Unmounting device...
umount: /dev/sdc: not mounted
[+] Zeroing out first chunk of device...
4096+0 records in
4096+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 0.848001 s, 19.8 MB/s
[+] Formatting /dev/sdc ...
start=0, blocks=8, type=RESERVED 
start=8, blocks=3, type=VRS 
start=11, blocks=245, type=USPACE 
start=256, blocks=1, type=ANCHOR 
start=257, blocks=16, type=PVDS 
start=273, blocks=1, type=LVID 
start=274, blocks=976756758, type=PSPACE 
start=976757032, blocks=1, type=ANCHOR 
start=976757033, blocks=239, type=USPACE 
start=976757272, blocks=16, type=RVDS 
start=976757288, blocks=1, type=ANCHOR 
[+] Writing fake MBR...
16+0 records in
16+0 records out
16 bytes copied, 0.000415576 s, 38.5 kB/s
2+0 records in
2+0 records out
2 bytes copied, 0.0398515 s, 0.1 kB/s
[+] Successfully formatted /dev/sdc: PTTYPE="dos"
Please disconnect/reconnect your drive now.
taohansen@nohostname:~/Documents> ./format-udf.sh sdc Morrow
[+] Testing dependencies...
[+] Looking for drive listing tool...which: no diskutil in (/usr/local/bin:/usr/bin:/bin:/usr/games:/sbin)
 using /sbin/blockdev
[+] Looking for unmount tool...which: no diskutil in (/usr/local/bin:/usr/bin:/bin:/usr/games:/sbin)
 using /usr/bin/umount
[+] Looking for UDF tool...which: no newfs_udf in (/usr/local/bin:/usr/bin:/bin:/usr/games:/sbin)
 using /usr/bin/mkudffs
[+] Parsing options...
[+] Validating arguments...
[+] Gathering drive information...
/dev/sdc: PTTYPE="dos"
BUP Fast HDD    
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw  1024   512   512          0   4000797859328   /dev/sdc
rw  1024   512   512          0    500099731968   /dev/sdc1
The above-listed device (and partitions, if any) will be completely erased.
Type 'yes' if this is what you intend:  yes
[+] Detecting total size...
[*] Using total size of 4000797859328
[+] Validating detected total size...
[+] Detecting physical block size...
[*] Using block size of 4096
[+] Validating detected block size...
[+] Unmounting device...
umount: /dev/sdc: not mounted
[+] Zeroing out first chunk of device...
4096+0 records in
4096+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 0.872517 s, 19.2 MB/s
[+] Formatting /dev/sdc ...
start=0, blocks=8, type=RESERVED 
start=8, blocks=3, type=VRS 
start=11, blocks=245, type=USPACE 
start=256, blocks=1, type=ANCHOR 
start=257, blocks=16, type=PVDS 
start=273, blocks=1, type=LVID 
start=274, blocks=976756758, type=PSPACE 
start=976757032, blocks=1, type=ANCHOR 
start=976757033, blocks=239, type=USPACE 
start=976757272, blocks=16, type=RVDS 
start=976757288, blocks=1, type=ANCHOR 
[+] Writing fake MBR...
16+0 records in
16+0 records out
16 bytes copied, 0.000392997 s, 40.7 kB/s
2+0 records in
2+0 records out
2 bytes copied, 0.000148946 s, 13.4 kB/s
[+] Successfully formatted /dev/sdc: PTTYPE="dos"
Please disconnect/reconnect your drive now.
@JElchison
Copy link
Owner

JElchison commented Oct 8, 2016

Hi @taohansen. Thanks for your note. Without being able to reproduce your issue myself, it's hard to conclude with certainly exactly what's going on. However, I've got some pretty good guesses.

If you update the set line at the top of the script to be:

set -eufx -o pipefail

...then you can see exactly what the script is doing behind the scenes.

Based on the output you provided, we can say with relative confidence that following is the line being executed by the script:

sudo mkudffs --utf8 --blocksize=4096 --udfrev=0x0201 --lvid="Morrow" --vid="Morrow" --media-type=hd /dev/sdc

Note how this differs with your control experiment:

mkudffs --media-type=hd --vid=Morrow --lvid=Morrow /dev/sdc

The script specifies the following parameters, which you omit:

       --utf8 Use UTF-8 for file names
       -b,--blocksize= block-size
              Specify the size of blocks in bytes. Valid block  size  vales  are
              1024,  2048  and  4096 bytes per block. If omitted, mkudffs block-
              size is 2048.
       -r,--udfrev= udf-revision
              Specify the udf revision  to  use.  Valid  revisions  are  0x0201,
              0x0200,  0x0150,  and  0x0102. If omitted, mkudffs udf-revision is
              0x0201.

--utf8 is likely inconsequential, and --udfrev defaults to the same value used by the script. I think the difference is that your control test used the default --blocksize of 2048. According to the script output, the script detected that your hard drive has a physical block size of 4096.

I'm willing to bet that if you tried your control test again, but this time explicitly setting --blocksize=4096, then you'll end up with the same result as with the script (i.e. failing to auto-mount in GNOME).

It's important to specify the correct block size for your device because UDF limits your partition size to 2 TiB with a 512 B block size, or 8 TiB with a 2048 B block size. (Essentially, that's a limit of 2^32 blocks for a UDF partition.) Also, the block size you specify may have performance implications on your specific hard drive. Only one value is correct, and using any incorrect value will slow down write operations (perhaps reads as well, depending on drive caching).

According to the script output, your disk seems to be 4 TB. This means that defaulting to a block size of 2048 should be fine in your case, except for the performance implications of using a non-matching block size.

In many cases, all you need to do is to manually specify the block size when mounting on Linux. For example:

mount -t udf -o bs=4096 /dev/sdc /mnt

If you're interested in learning more about the landscape, here is some additional reading:

Let me know how it goes!

In the unlikely case that you're running into #13 and its corresponding Linux bug https://bugzilla.kernel.org/show_bug.cgi?id=102271 (meaning that 4096 actually isn't the correct value for your drive), then you can use format-udf's -b switch to override the detected value:

    -b BLOCK_SIZE
        Block size to be used during format operation.
        If absent, defaults to value reported by blockdev/diskutil.
        This is useful in light of the following Linux kernel bug:
            https://bugzilla.kernel.org/show_bug.cgi?id=102271
        See also:
            https://github.com/JElchison/format-udf/issues/13

@ghost
Copy link
Author

ghost commented Oct 9, 2016

Thank you for taking the time to write such a detailed answer.

I ran your script again and then sudo mount -t udf -o bs=4096 /dev/sdc /mnt which failed to mount the drive in userspace.

So then I combed through older issues and tried again first by running sudo mkdir /media && sudo mkdir /media/Morrow then sudo mount -t udf -o bs=4096 /dev/sdc /media/Morrow/ which gave me a mounted drive accessible through GNOME Files.

Overcome with joy, I plugged the drive in to my friend's Mac, which resulted in this error: The disk you inserted was not readable by this computer with an option to initialize or eject the drive.

So I now have a drive which I can't immediately test with a Windows machine but which does not mount with Mac defeating my desired use for UDF.

So I now have three questions:
How do I fix this? Why does my distro refuse to auto-mount a blocksize 4096 drive by default? How can I change this behavior?

@JElchison
Copy link
Owner

Hi again, @taohansen.

It looks like you stumbled into the correct line of reasoning. As with any mount command in Linux, you must specify an empty directory. It sounds like /mnt wasn't an empty directory on your system, which is why it failed to mount. Once you created/specified an empty directory (/media/Morrow), it appears that your mount attempt succeeded.

In the same way that you had to specify manual instructions for Linux to mount your 4K-block drive, you'll likely have to do the same on OS X. See #16 for more info. To summarize, this might work for you to manually mount your new UDF drive on OS X:

sudo mount_udf -b 4096 /dev/diskX /Volumes/Morrow

Of course, you'll have to fill in the correct X. Keep in mind that /Volumes/Morrow must be an empty directory, just like on Linux.

All of this is required because operating systems like openSUSE and OS X don't (usually) perform physical block detection when attempting to auto-mount media. I have to do the same thing for my 4K-block drive as well. It's a small price to pay for a drive (with modern features) that can be used across OS families.

Good luck!

@JElchison
Copy link
Owner

FYI, I'm closing this issue with the clarifying text I've just added on the README. Essentially, there's not much that format-udf can do if Linux and OS X have trouble auto-mounting drives with certain block sizes.

Please feel free to continue the conversation by commenting below. I'm still happy to assist however I can.

@ghost
Copy link
Author

ghost commented Oct 9, 2016

sudo mount_udf -b 4096 -v all /dev/disk1 /Volumes/Morrow gives the error:

mount_udf: mount_udf: /dev/disk1 failed, Resource busy, Resource busy

@JElchison
Copy link
Owner

is disk1 the proper identifier for your disk? You can use diskutil list and diskutil info to learn more.

Assuming it's correct, you won't be able to mount a drive that's already mounted. Check the output from mount to verify drive mount status.

@ghost
Copy link
Author

ghost commented Oct 9, 2016

Aha, this is same result of me mounting to /mnt/ in Linux: under macOS, the drive fails to show up in the Finder but I can cd to it just fine. Is there any way to mount the drive so Finder sees it just as I was able to modify the mount location to /media/ under Linux and see it in GNOME Files?

@JElchison
Copy link
Owner

There are two requirements mounting (same on Linux as for OS X):

  1. Provide a path to an empty directory and the proper permissions
  2. Provide the proper block size when mounting

If you do neither, you'll see an empty directory and no listing in the output from mount.

If you do 1 but not 2, you'll see an empty directory but it might be listed in the output from mount (also likely an error message).

You didn't provide enough data for me to know what scenario you're in, but there's nothing special about the Finder ... if you mount it properly at the CLI, then it will show up in the Finder.

@ghost
Copy link
Author

ghost commented Oct 9, 2016

I am happy to run whatever commands you specify to help resolve this issue. I have changed the permissions of the folder /Volumes/Morrow I created and replugged the drive but am receiving the same error. I have specified the proper blocksize.

I was hopeful for a plug & play universal solution but it is as difficult for me to install the Paragon NTFS driver as it is to mount this drive and a drive formatted under NTFS in tandem with Paragon's driver will automount under these three operating systems. That only UDF 512 blocksize drives will mount at all under Windows is the arrow in the heart for UDF for me. I thank you for your exhaustive help and apologize if I've been at all untoward.

@JElchison
Copy link
Owner

Hi @taohansen. Just to be clear, I'm more than happy to help you with format-udf.sh, but I can't help you change OS X or Windows (correct me if I'm wrong, but that seems to be what you're asking for). Apple and Microsoft are the ones in control of how their OSes (auto)mount UDF drives. Microsoft has no motivation to add robust support for UDF, as they prefer that we use NTFS to begin with. Same with Apple and HFS+.

I'm happy to give you explicit commands, but you still haven't shared what your block size is. :) format-udf detected it as 4096, but you never reported back as to whether you could be running into #13 (i.e. the Linux bug that lies about your block size for AF drives). Please see https://bugzilla.kernel.org/show_bug.cgi?id=102271. Have you tried hdparm or smartctl on Linux? Does the output from diskutil info on OS X corroborate how Linux detected it? Do you have any documentation from the hard drive vendor (or diagnostic info from proprietary tools) that specifies exactly what the logical and physical block sizes are?

As you can see, this is a complicated landscape. :) There is no "plug & play universal solution", but UDF is the closest we've got in the open source arena. Closed-source formats/drivers present challenges of their own. If you're more comfortable with that path, then I certainly won't be offended. In fact, if you're okay with a proprietary solution, I might recommend that altogether. At the very least, NTFS has a fsck, and UDF doesn't. Many people view that as a non-starter for UDF (I just had a corrupt UDF filesystem last week, and had to format and start over).

Let me know how you'd like to proceed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant