Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

recognize multisession UDF optical discs (CD / DVD) #836

Open
md-work opened this issue Jan 15, 2021 · 31 comments
Open

recognize multisession UDF optical discs (CD / DVD) #836

md-work opened this issue Jan 15, 2021 · 31 comments

Comments

@md-work
Copy link

md-work commented Jan 15, 2021

Please add support to recognize UDF multisession optical discs.
 

udisks currently doesn't seem to recognize UDF multisession optical discs (CD / DVD).
Tested with openSUSE-15.2 (udisks2-2.8.1) and Debian-11-Testing (Bullseye 2021-01-14 - udisks2-2.9.1)

Instead udisks either recognizes the disc as ISO-9660 (if an ISO-9660 filesystem exists) or simply detects no filesystem at all.
 
 

Details

To be precise: It must be detected what filesystem the LAST session of a disc has.
(for example blkid from util-linux instead somehow always detects the filesystem of the first session)

https://github.com/pali/udftools seems to have a working implementation to detect if the last session has an UDF filesystem. Just run udfinfo or udflabel on a disc.
 
 

Reproduction

I made some CD images to reproduce the problem.
udisks_udf_multisession_support.zip

Sadly multisession CDs can't simply be stored in an dd/ISO image. Instead I've created some images which can be used via CDemu (Wikipedia).
Each image consists of a *.bin and a *.toc file for each session. Loading a full CD with all sessions works like this:

cdemu  load  0  image_session-1.toc  image_session-2.toc
@md-work
Copy link
Author

md-work commented Feb 5, 2021

If I mount the device another bug is triggered on the dbus interface.
mount -o ro /dev/sr0 /mnt
Usually there would be a PropertiesChanged signal on the dbus interface. But instead there's an InterfacesAdded signal.

And on unmount this bug continues.
umount /mnt
Usually this would trigger another PropertiesChanged signal in the dbus interface. But instead now there's an InterfacesRemoved.

@tbzatek
Copy link
Member

tbzatek commented Feb 9, 2021

You can assume that both the InterfacesAdded and PropertiesChanged signals are valid and can be received in random order, as a result of asynchronous nature of the messaging chain. What's your use case here?

As for the multisession, udisks relies on information from udev which in turn often relies on information from blkid. In case blkid is wrong, please open a bugreport on https://github.com/karelzak/util-linux/ and udisks should pick the information up automatically.

@tbzatek
Copy link
Member

tbzatek commented Feb 9, 2021

Also, could you post udevadm info /dev/....., blkid and lsblk dumps here related to the block device carrying the UDF filesystem please?

@md-work
Copy link
Author

md-work commented Feb 9, 2021

@tbzatek

You can assume that both the InterfacesAdded and PropertiesChanged signals are valid and can be received in random order, as a result of asynchronous nature of the messaging chain. What's your use case here?

I'm listening to these signals in a software which needs to know the currently connected storage devices.
Don't get me wrong. In general this bug affects probably every software using udisks. So for example KDE/Plasma isn't able to mount UDF multisession discs.

I can workaround this behavior in my software. But I'm mentioning this, because the behavior differs between UDF-multisession discs and non UDF/multisession discs (everything is fine for these). So I'm considering this to be a bug in udisks and I guess it should better be fixed in udisks (especially the wrongly InterfacesRemoved signal on umount).

As for the multisession, udisks relies on information from udev which in turn often relies on information from blkid.

As far as I know blkid doesn't provide information about mounts. And mounts clearly play a role here.
But I'll talk about blkid later on.

 

Though I guess this might be subsequent faults of the initially described problem. So let's not focus on this signaling problem for the moment.

 

 


 

 

Also, could you post udevadm info /dev/....., blkid and lsblk dumps here related to the block device carrying the UDF filesystem please?

Sure! (see below)

Nevertheless, please feel free to run your own tests with the provided disc images and CDemu.
It should be at least reproducible with openSUSE-15.2 and Debian-11-Testing (Bullseye 2021-01-14).

I've reproduced the same results with a real DVD instead of a virtual cdemu disc.
Sadly the only way to create pure UDF multisession discs I know is CDBurnerXP for Windows.
So these cdemu compatible images should be fine for debugging.

Everything done as root for simplicity of testing.
Prerequisite: Load & start the cdemu kernel moduel and daemon modprobe vhba; cdemu-daemon

 

used programs

All by openSUSE-15.2, except udfinfo which was compiled from version 2.3 source.

used program details
$ cat /etc/os-release 
NAME="openSUSE Leap"
VERSION="15.2"
ID="opensuse-leap"
ID_LIKE="suse opensuse"
VERSION_ID="15.2"
PRETTY_NAME="openSUSE Leap 15.2"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:leap:15.2"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"

$ cdemu version
Library version: 3.2.1
Daemon version: 3.1.0

$ rpm -qf /usr/bin/udisksctl 
udisks2-2.8.1-lp152.1.4.x86_64

$ rpm -qf /usr/bin/udevadm 
udev-234-lp152.31.19.1.x86_64

$ lsblk --version
lsblk from util-linux 2.33.1

$ udfinfo --help
udfinfo from udftools 2.3
Usage:
        udfinfo [--locale|--u8|--u16|--utf8] [-b|--blocksize=block-size] [--startblock=block] [--lastblock=block] [--vatblock=block] device

 

 

UDF non-multisession example (everything's fine)

Loading session 1 only, so this isn't multisession.

non-multisession dump (working fine)
$ cdemu unload 0; sleep 2; cdemu load 0 MINI_-_UDF-MulSess_1.toc

$ udisksctl info -b /dev/sr1
/org/freedesktop/UDisks2/block_devices/sr1:
  org.freedesktop.UDisks2.Block:
    Configuration:              []
    CryptoBackingDevice:        '/'
    Device:                     /dev/sr1
    DeviceNumber:               2817
    Drive:                      '/org/freedesktop/UDisks2/drives/CDEmu_Virt_2e_CD_2fDVD_ROM'
    HintAuto:                   true
    HintIconName:               
    HintIgnore:                 false
    HintName:                   
    HintPartitionable:          false
    HintSymbolicIconName:       
    HintSystem:                 false
    Id:                         by-uuid-2010112200000000
    IdLabel:                    UDF-MulSess_1
    IdType:                     udf
    IdUUID:                     2010112200000000
    IdUsage:                    filesystem
    IdVersion:                  1.02
    MDRaid:                     '/'
    MDRaidMember:               '/'
    PreferredDevice:            /dev/sr1
    ReadOnly:                   true
    Size:                       1249280
    Symlinks:                   /dev/disk/by-label/UDF-MulSess_1
                                /dev/disk/by-path/platform-vhba-scsi-0:0:1:0
                                /dev/disk/by-uuid/2010112200000000
    UserspaceMountOptions:      
  org.freedesktop.UDisks2.Filesystem:
    MountPoints:        
    Size:               0

$ udevadm info /dev/sr1                       
P: /devices/platform/vhba/host3/target3:0:1/3:0:1:0/block/sr1
N: sr1
S: disk/by-label/UDF-MulSess_1
S: disk/by-path/platform-vhba-scsi-0:0:1:0
S: disk/by-uuid/2010112200000000
E: DEVLINKS=/dev/disk/by-label/UDF-MulSess_1 /dev/disk/by-path/platform-vhba-scsi-0:0:1:0 /dev/disk/by-uuid/2010112200000000
E: DEVNAME=/dev/sr1
E: DEVPATH=/devices/platform/vhba/host3/target3:0:1/3:0:1:0/block/sr1
E: DEVTYPE=disk
E: ID_BUS=scsi
E: ID_CDROM=1
E: ID_CDROM_CD=1
E: ID_CDROM_CD_R=1
E: ID_CDROM_CD_RW=1
E: ID_CDROM_DVD=1
E: ID_CDROM_DVD_PLUS_R=1
E: ID_CDROM_DVD_R=1
E: ID_CDROM_DVD_RAM=1
E: ID_CDROM_MEDIA=1
E: ID_CDROM_MEDIA_CD=1
E: ID_CDROM_MEDIA_SESSION_COUNT=1
E: ID_CDROM_MEDIA_TRACK_COUNT=1
E: ID_CDROM_MEDIA_TRACK_COUNT_DATA=1
E: ID_CDROM_MRW=1
E: ID_CDROM_MRW_W=1
E: ID_FOR_SEAT=block-platform-vhba-scsi-0_0_1_0
E: ID_FS_LABEL=UDF-MulSess_1
E: ID_FS_LABEL_ENC=UDF-MulSess_1
E: ID_FS_TYPE=udf
E: ID_FS_USAGE=filesystem
E: ID_FS_UUID=2010112200000000
E: ID_FS_UUID_ENC=2010112200000000
E: ID_FS_VERSION=1.02
E: ID_MODEL=Virt._CD_DVD-ROM
E: ID_MODEL_ENC=Virt.\x20CD\x2fDVD-ROM
E: ID_PATH=platform-vhba-scsi-0:0:1:0
E: ID_PATH_TAG=platform-vhba-scsi-0_0_1_0
E: ID_REVISION=1.10
E: ID_SCSI=1
E: ID_SCSI_INQUIRY=1
E: ID_TYPE=cd
E: ID_VENDOR=CDEmu
E: ID_VENDOR_ENC=CDEmu\x20\x20\x20
E: MAJOR=11
E: MINOR=1
E: SCSI_MODEL=Virt._CD/DVD-ROM
E: SCSI_MODEL_ENC=Virt.\x20CD/DVD-ROM
E: SCSI_REVISION=1.10
E: SCSI_TPGS=0
E: SCSI_TYPE=cd/dvd
E: SCSI_VENDOR=CDEmu
E: SCSI_VENDOR_ENC=CDEmu\x20\x20\x20
E: SUBSYSTEM=block
E: SYSTEMD_MOUNT_DEVICE_BOUND=1
E: TAGS=:systemd:uaccess:seat:
E: USEC_INITIALIZED=76936869

$ blkid /dev/sr1 
/dev/sr1: UUID="2010112200000000" LABEL="UDF-MulSess_1" TYPE="udf"

$ lsblk /dev/sr1 
NAME MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr1   11:1    1  1,2M  0 rom

$ ~/opt/udftools/bin/udfinfo /dev/sr1 
filename=/dev/sr1
label=UDF-MulSess_1
uuid=2010112200000000
lvid=UDF-MulSess_1
vid=UDF-MulSess_1
vsid=
fsid=UDF-MulSess_1
fullvsid=20101122
owner=
organization=
contact=
appid=
impid=*UDF Image Creator
winserialnum=0xbd061f8f
blocksize=2048
blocks=610
usedblocks=6
freeblocks=0
behindblocks=256
numfiles=1
numdirs=1
udfrev=1.02
udfwriterev=1.02
lastblock=610
integrity=closed
accesstype=readonly
softwriteprotect=no
hardwriteprotect=no
start=16, blocks=3, type=VRS
start=32, blocks=16, type=MVDS
start=64, blocks=1, type=LVID
start=128, blocks=16, type=RVDS
start=256, blocks=1, type=ANCHOR
start=320, blocks=6, type=PSPACE
start=353, blocks=1, type=ANCHOR

 

 

UDF multisession example (broken)

Loading session 1 and 2, so this appears as a multisession disc.

Most information by udisksctl is completly wrong (except the size).

blkid is wrong in an interesting way, as it shows the information of the first session.
(I labeled the first session UDF-MulSess_1 and second session UDF-MulSess_2 to make them easily distinguishable)
This looks like the related bug for blkid: util-linux/util-linux#1161

udfinfo is correct, as it correctly shows the second sessions label UDF-MulSess_2 and other details of the second session.

udevadm and lsblk seem OK. But they just doesn't report the information essential to me (the filesystem).

multisession dump (broken)
$ cdemu unload 0; sleep 2; cdemu load 0 MINI_-_UDF-MulSess_1.toc MINI_-_UDF-MulSess_2.toc

$  udisksctl info -b /dev/sr1
/org/freedesktop/UDisks2/block_devices/sr1:
  org.freedesktop.UDisks2.Block:
    Configuration:              []
    CryptoBackingDevice:        '/'
    Device:                     /dev/sr1
    DeviceNumber:               2817
    Drive:                      '/org/freedesktop/UDisks2/drives/CDEmu_Virt_2e_CD_2fDVD_ROM'
    HintAuto:                   true
    HintIconName:               
    HintIgnore:                 false
    HintName:                   
    HintPartitionable:          false
    HintSymbolicIconName:       
    HintSystem:                 false
    Id:                         
    IdLabel:                    
    IdType:                     
    IdUUID:                     
    IdUsage:                    
    IdVersion:                  
    MDRaid:                     '/'
    MDRaidMember:               '/'
    PreferredDevice:            /dev/sr1
    ReadOnly:                   true
    Size:                       25845760
    Symlinks:                   /dev/disk/by-path/platform-vhba-scsi-0:0:1:0
    UserspaceMountOptions:

$ udevadm info /dev/sr1
P: /devices/platform/vhba/host3/target3:0:1/3:0:1:0/block/sr1
N: sr1
S: disk/by-path/platform-vhba-scsi-0:0:1:0
E: DEVLINKS=/dev/disk/by-path/platform-vhba-scsi-0:0:1:0
E: DEVNAME=/dev/sr1
E: DEVPATH=/devices/platform/vhba/host3/target3:0:1/3:0:1:0/block/sr1
E: DEVTYPE=disk
E: ID_BUS=scsi
E: ID_CDROM=1
E: ID_CDROM_CD=1
E: ID_CDROM_CD_R=1
E: ID_CDROM_CD_RW=1
E: ID_CDROM_DVD=1
E: ID_CDROM_DVD_PLUS_R=1
E: ID_CDROM_DVD_R=1
E: ID_CDROM_DVD_RAM=1
E: ID_CDROM_MEDIA=1
E: ID_CDROM_MEDIA_CD=1
E: ID_CDROM_MEDIA_SESSION_COUNT=2
E: ID_CDROM_MEDIA_SESSION_LAST_OFFSET=24596480
E: ID_CDROM_MEDIA_TRACK_COUNT=2
E: ID_CDROM_MEDIA_TRACK_COUNT_DATA=2
E: ID_CDROM_MRW=1
E: ID_CDROM_MRW_W=1
E: ID_FOR_SEAT=block-platform-vhba-scsi-0_0_1_0
E: ID_MODEL=Virt._CD_DVD-ROM
E: ID_MODEL_ENC=Virt.\x20CD\x2fDVD-ROM
E: ID_PATH=platform-vhba-scsi-0:0:1:0
E: ID_PATH_TAG=platform-vhba-scsi-0_0_1_0
E: ID_REVISION=1.10
E: ID_SCSI=1
E: ID_SCSI_INQUIRY=1
E: ID_TYPE=cd
E: ID_VENDOR=CDEmu
E: ID_VENDOR_ENC=CDEmu\x20\x20\x20
E: MAJOR=11
E: MINOR=1
E: SCSI_MODEL=Virt._CD/DVD-ROM
E: SCSI_MODEL_ENC=Virt.\x20CD/DVD-ROM
E: SCSI_REVISION=1.10
E: SCSI_TPGS=0
E: SCSI_TYPE=cd/dvd
E: SCSI_VENDOR=CDEmu
E: SCSI_VENDOR_ENC=CDEmu\x20\x20\x20
E: SUBSYSTEM=block
E: SYSTEMD_MOUNT_DEVICE_BOUND=1
E: TAGS=:uaccess:seat:systemd:
E: USEC_INITIALIZED=76936869

$ blkid /dev/sr1 
/dev/sr1: UUID="2010112200000000" LABEL="UDF-MulSess_1" TYPE="udf"

$ lsblk /dev/sr1 
NAME MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr1   11:1    1 24,7M  0 rom

$ ~/opt/udftools/bin/udfinfo /dev/sr1 
udfinfo: Warning: First and third Anchor Volume Descriptor Pointer not found, using second
filename=/dev/sr1
label=UDF-MulSess_2
uuid=2010112200000000
lvid=UDF-MulSess_2
vid=UDF-MulSess_2
vsid=
fsid=UDF-MulSess_2
fullvsid=20101122
owner=
organization=
contact=
appid=
impid=*UDF Image Creator
winserialnum=0xbd08075b
blocksize=2048
blocks=12620
usedblocks=12018
freeblocks=0
behindblocks=256
numfiles=2
numdirs=1
udfrev=1.02
udfwriterev=1.02
startblock=12010
lastblock=12620
integrity=closed
accesstype=readonly
softwriteprotect=no
hardwriteprotect=no
start=320, blocks=12018, type=PSPACE
start=12026, blocks=3, type=VRS
start=12042, blocks=16, type=MVDS
start=12074, blocks=1, type=LVID
start=12138, blocks=16, type=RVDS
start=12363, blocks=1, type=ANCHOR

@md-work
Copy link
Author

md-work commented Feb 11, 2021

Maybe related: systemd/systemd#17780

@pali
Copy link

pali commented Feb 20, 2021

I fixed both ISO and UDF superblock modules in libblkid to read last session from multisession optical disc:

But new version of util-linux with these fixes were not released yet. Note that without fixes blkid is not able to read multisession UDF disc. For testing you could try to compile util-linux from master git branch, blkid should properly detects multisession ISO and UDF discs.

I have also fixed udfinfo and udflabel in udftools 2.3 to properly handle multisession optical discs. You need to update udftools to version 2.3.

@pali
Copy link

pali commented Feb 21, 2021

$ cdemu unload 0; sleep 2; cdemu load 0 MINI_-_UDF-MulSess_1.toc MINI_-_UDF-MulSess_2.toc
...
$ ~/opt/udftools/bin/udfinfo /dev/sr1 
udfinfo: Warning: First and third Anchor Volume Descriptor Pointer not found, using second

Either UDF disc is corrupted or this looks like a bug in udfinfo. First Anchor Volume Descriptor Pointer should be available.

@md-work: Could you please provide me image of the /dev/sr1? E.g. generated by dd if=/dev/sr1 of=image.udf && gzip -9 image.udf.

By the way, udfinfo is very smart and can read Anchor Volume Descriptor Pointer (superblock of UDF) also from backup location, but this does not have to be supported by other UDF implementations (kernel, blkid, ...).

filename=/dev/sr1
label=UDF-MulSess_2
uuid=2010112200000000
lvid=UDF-MulSess_2
vid=UDF-MulSess_2
vsid=
fsid=UDF-MulSess_2
fullvsid=20101122

@md-work
Copy link
Author

md-work commented Feb 22, 2021

$ cdemu unload 0; sleep 2; cdemu load 0 MINI_-_UDF-MulSess_1.toc MINI_-_UDF-MulSess_2.toc
...
$ ~/opt/udftools/bin/udfinfo /dev/sr1 
udfinfo: Warning: First and third Anchor Volume Descriptor Pointer not found, using second

Either UDF disc is corrupted or this looks like a bug in udfinfo. First Anchor Volume Descriptor Pointer should be available.

I get the same "udfinfo" warning for a real disc without using cdemu.
But the disc is being mounted without problems by the Linux kernel.

 

@md-work: Could you please provide me image of the /dev/sr1? E.g. generated by dd if=/dev/sr1 of=image.udf && gzip -9 image.udf.

Are you aware, that "dd" will only read the first session only!?
This is why I used "cdrdao" before (it can properly image all sessions).

Also "dd" will exit with code 1 at the end of the block device. So gzip will never be executed.

If you're still sure that you want dd if=/dev/sr0 of=image.udf && gzip -9 image.udf I can gladly do that.

In case, I guess it shall be a real disc and not something loaded into cdemu!?
I don't have the MINI_-_UDF-MulSess original disc anymore (just the image). But I have another real disc. It's the one I just tested "udfinfo" with (see above section).

 

By the way, udfinfo is very smart and can read Anchor Volume Descriptor Pointer (superblock of UDF) also from backup location, but this does not have to be supported by other UDF implementations (kernel, blkid, ...).

It would be nice if udisk would behave mostly like the Linux kernel.
Or at least, if Linux can mount a disc udisks should be able to read the disc too.

@pali
Copy link

pali commented Feb 22, 2021

I get the same "udfinfo" warning for a real disc without using cdemu.
But the disc is being mounted without problems by the Linux kernel.

That is why I wan to see that image as it can be a bug in udfinfo.

Are you aware, that "dd" will only read the first session only!?

dd read all data sessions (concatenated) with correct block numbering. It only does not provide ToC (information on which block first session ends and on which second session starts) and non-data sessions (e.g. audio), but this is not not our case.

Also "dd" will exit with code 1 at the end of the block device. So gzip will never be executed.

If you're still sure that you want dd if=/dev/sr0 of=image.udf && gzip -9 image.udf I can gladly do that.

So dd if=/dev/sr0 of=image.udf; gzip -9 image.udf then.

It would be nice if udisk would behave mostly like the Linux kernel.
Or at least, if Linux can mount a disc udisks should be able to read the disc too.

This is feature request for udisks, I will let it for udisks developers.

@md-work
Copy link
Author

md-work commented Feb 23, 2021

Are you aware, that "dd" will only read the first session only!?

dd read all data sessions (concatenated) with correct block numbering. It only does not provide ToC (information on which block first session ends and on which second session starts) and non-data sessions (e.g. audio), but this is not not our case.

Here is the requested image (it's the disc I tested udfinfo with in my last comment):
image_11MB_2-sessions.udf.gz

But as far as I can see it only contains files from the first session.
On session two there are files over 11 MB in size. But the uncompressed dd image is only less than 2 MB in size.

$ dd if=/dev/sr0 of=image.udf; gzip -9 image.udf
dd: error reading '/dev/sr0': Input/output error
2432+0 records in
2432+0 records out
1245184 bytes (1.2 MB, 1.2 MiB) copied, 5.45237 s, 228 kB/s
$ lsblk /dev/sr0
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0   11:0    1  35M  0 rom  
$ mount -o ro /dev/sr0 /mnt
$ ls -lhs /mnt/
total 11M
5.2M -r--r--r-- 1 nobody nobody 5.2M Jan 11 17:58 cdbxp_setup_4.5.8.7128_minimal_-_session_2_only.exe
5.2M -r--r--r-- 1 nobody nobody 5.2M Jan 11 20:22 isobuster_install_-_4.7isobuster_install_-_4.7_-_session_2_only.exe
2.0K -r--r--r-- 1 nobody nobody   34 Jan 11 21:09 session_1_and_2.txt
2.0K -r--r--r-- 1 nobody nobody   33 Jan 11 19:14 session_2_only.txt

Here's the cdrdao image from the same physical disc with full 11 MB of size for comparison:
image_11MB_2-sessions_-_cdrdao.zip

@pali
Copy link

pali commented Feb 23, 2021

But as far as I can see it only contains files from the first session.

Looks like that your cdrom emulation software creates holes in /dev/sr0 between sessions and when trying to read these holes that cdrom emulation software returns -EIO (Input/output error) instead of returning zeros, which makes dd to immetiately stop.

Here's the cdrdao image from the same physical disc with full 11 MB of size for comparison:
image_11MB_2-sessions_-_cdrdao.zip

You have put sessions into separate files. Such files are unusable for any UDF software as UDF filesystem is addressed by absolute block positions. So I need one continous file like it can be accessible by opening and reading /dev/sr0.

Could you provide just one data image file (either by cdrdao or any other software)? I do not need metadata (like ToC).
I think that conv=noerror option for dd could instruct dd to skip errors. Or there is also ddrescue.

@md-work
Copy link
Author

md-work commented Feb 23, 2021

Looks like that your cdrom emulation software creates holes in /dev/sr0 between sessions and when trying to read these holes that cdrom emulation software returns -EIO (Input/output error) instead of returning zeros, which makes dd to immetiately stop.

I didn't use a cdrom software for that image. It's directly from a physical disc!

 

[...]
You have put sessions into separate files. Such files are unusable for any UDF software as UDF filesystem is addressed by absolute block positions. So I need one continous file like it can be accessible by opening and reading /dev/sr0.

cdrdao can't do multiple sessions in one file.

Could you provide just one data image file (either by cdrdao or any other software)? I do not need metadata (like ToC).
I think that conv=noerror option for dd could instruct dd to skip errors. Or there is also ddrescue.

I created another dd image with conv=noerror and it's now 35 MB in size.
image-noerror_11MB_2-sessions.udf.gz
I put a file with the content This file is on session 2 only on the second session and that string can indeed be found in that image (for example using strings command).

Nevertheless, Linux always mounts the first session only. Even when specifying mount -o session=2 ....
I guess Linux needs that explicit errors to know there's a second session. But the noerror option converts that errors into zeros, so Linux doesn't get the clue for the second session.

 

Actually I think you might be on the wrong track!
This is about sessions which is a feature outside the udf filesystem.
You might have a look into the code udfinfo is using to detect the data on the second session.

@pali
Copy link

pali commented Feb 23, 2021

I didn't use a cdrom software for that image. It's directly from a physical disc!

Ok! Also cdrom drive is free to return -EIO for some block of data.

cdrdao can't do multiple sessions in one file.

:-( hard to use this sw for debugging then.

I created another dd image with conv=noerror and it's now 35 MB in size.
image-noerror_11MB_2-sessions.udf.gz

Thank you! I will look at it and investiagate it.

Nevertheless, Linux always mounts the first session only. Even when specifying mount -o session=2 ....

Actually I think you might be on the wrong track!
This is about sessions which is a feature outside the udf filesystem.
You might have a look into the code udfinfo is using to detect the data on the second session.

I do not think so. Session borders are invisible in block device. block device does not see them. If you want to get information on which block is starting which session you need to use out-of-band data. For cd rom driver (/dev/sr0) you need to retrieve it e.g. via CDROMMULTISESSION ioctl. And both kernel and udfinfo is doing it. Kernel's udf driver by default read last session, but it only works when mounting /dev/sr0 (which provides CDROMMULTISESSION ioctl).

@pali
Copy link

pali commented Feb 23, 2021

Could you provide also output from /usr/bin/cd_sessions /dev/sr0 for that physical cdrom image?

@pali
Copy link

pali commented Feb 23, 2021

I created another dd image with conv=noerror and it's now 35 MB in size.
image-noerror_11MB_2-sessions.udf.gz

This image is unusable... it looks like that dd omitted empty blocks instead of filling zeros. Could you try to use other tool? e.g. ddrescue?

@md-work
Copy link
Author

md-work commented Feb 23, 2021

Could you provide also output from /usr/bin/cd_sessions /dev/sr0 for that physical cdrom image?
Used: http://www.13thmonkey.org/udfclient/releases/UDFclient.0.8.11.tgz

cd_sessions /dev/sr0 
Opening device /dev/sr0
Device has MMC profile 0x09
Disc info for disc in device /dev/sr0
        MMC profile        : CD-R recordable
        sequential         : yes
        recordable         : yes
        erasable           :  no
        blankable          :  no
        formattable        :  no
        rewritable         :  no
        mount raineer      :  no
        packet writing     :  no
        strict overwrite   :  no
        blocking number    : 1
        disc state         : incomplete (appendable)
        last session state : empty
        sectorsize         : 2048
        Number of sessions     3
        Session 0
                start  at         0
                ends   at         608
                length for        608
                next writable at  0
                free blocks       0
                packet size       0

        Session 1
                start  at         12010
                ends   at         17898
                length for        5888
                next writable at  0
                free blocks       0
                packet size       0

        Session 2
                start  at         24800
                ends   at         359845
                length for        335045
                next writable at  24800
                free blocks       335045
                packet size       0

Disc access statistics
        sector reads          0  (0 Kbyte)
        sector written        0  (0 Kbyte)
        switches              0

Interesting, although I added files to the disc just once there's a third "virtual!?" session!
I created another cdrdao image and also extracted the 3rd session this time:
image_11MB_2plus1-sessions_-_cdrdao.zip

cdrdao even didn't complain when trying to read a 4th and 5th session. But the resulting *.bin files where identical with the 3rd session. So I guess those 4th and 5th sessions don't really exist.

 

 

I created another dd image with conv=noerror and it's now 35 MB in size.
image-noerror_11MB_2-sessions.udf.gz

This image is unusable... it looks like that dd omitted empty blocks instead of filling zeros. Could you try to use other tool? e.g. ddrescue?

I'm not sure how to achieve this.
Can you give me a hint to follow?

Couldn't you further investigate this by loading the cdrdao image I provided into cdemu?
On my machine this triggers exactly the same problem as the physical disc does!

If you need a single image, you could use dd or ddrescue on the cdemu drive.

 

Alternatively please try creating a physical disc for yourself.
I put a very detailed explanation on how I created that physical disc in the first place beside the first images I posted here.
udisks_udf_multisession_support.zip -> Multisession and UDF (CD and DVD).txt
Only downside: You'll probably need a computer with a CD/DVD burner and running Windows natively.

You can get the cd burning software (CDBurnerXP) setup here:
https://cdburnerxp.se/downloadsetup.exe
https://www.heise.de/download/product/cdburnerxp-35927

@pali
Copy link

pali commented Feb 23, 2021

Interesting, although I added files to the disc just once there's a third "virtual!?" session!

I think this is because you have not closed disc yet, so this third session is a place where new session can start. See next writable at.

I'm not sure how to achieve this. Can you give me a hint to follow?

I think that simple call /usr/bin/ddrescue /dev/sr0 image.udf should be enough. Other switches are described in manual page.

Only downside: You'll probably need a computer with a CD/DVD burner and running Windows natively.

And this is the issue, I do not have any native for a long time.

@md-work
Copy link
Author

md-work commented Feb 23, 2021

I'm not sure how to achieve this. Can you give me a hint to follow?

I think that simple call /usr/bin/ddrescue /dev/sr0 image.udf should be enough. Other switches are described in manual page.

image-ddrescue_11MB_2-sessions.udf.gz

$ ddrescue /dev/sr0 image-ddrescue_11MB_2-sessions.udf
GNU ddrescue 1.23
Press Ctrl-C to interrupt
     ipos:   24300 kB, non-trimmed:        0 B,  current rate:       0 B/s
     opos:   24300 kB, non-scraped:        0 B,  average rate:    197 kB/s
non-tried:        0 B,  bad-sector:    49152 B,    error rate:     512 B/s
  rescued:   36605 kB,   bad areas:        3,        run time:      3m  4s
pct rescued:   99.86%, read errors:      100,  remaining time:         n/a
                              time since last successful read:      1m 21s
Finished

Also tried dd_rescue (differs from ddrescue). But the output file was exactly the same.

 

Only downside: You'll probably need a computer with a CD/DVD burner and running Windows natively.

And this is the issue, I do not have any native for a long time.

I confess, I used my girlfriends notebook for that part because I don't have any native Windows too :-D

@pali
Copy link

pali commented Feb 23, 2021

Perfect, output from cd_sessions and file image-ddrescue_11MB_2-sessions.udf.gz seems to be usable! I will investigate it and let you know.

What I have learnt today? Do not use dd but rather ddrescue.

@md-work
Copy link
Author

md-work commented Feb 23, 2021

Definitely learned that too.
I though you need a session aware software to get all bits from a disc, but ddrescue seems to do the job too.

Nevertheless, I can't reproduce the exact udisks behavior of the physical disc when using the ddrescue image.
I tried the ddrescue image via cdemu, losetup and by passing it to a qemu vm.
In all cases udisks behaved like the disc had just the first session.

Only by using the cdrdao image via cdemu I can reproduce the udisks problem in the same manner as with the physical disc!
I guess this is because cdemu is able to provide suitable out-of-band-data.

@pali
Copy link

pali commented Feb 23, 2021

Nevertheless, I can't reproduce the exact udisks behavior of the physical disc when using the ddrescue image.

That is expected. Image itself does not contain session information. This is stored only in out-of-band data. But you have provided it in cd_sessions output, so I can "simulate" it (udftools have --startblock and --lastblock options).

@pali
Copy link

pali commented Feb 23, 2021

udfinfo: Warning: First and third Anchor Volume Descriptor Pointer not found, using second

And now I see where is the issue.

Lets read first session:

$ ./udfinfo/udfinfo -b 2048 --startblock=0 --lastblock=608 image-ddrescue_11MB_2-sessions.udf
filename=image-ddrescue_11MB_2-sessions.udf
label=UDF-MulSess_1
uuid=2010112200000000
lvid=UDF-MulSess_1
vid=UDF-MulSess_1
vsid=
fsid=UDF-MulSess_1
fullvsid=20101122
owner=
organization=
contact=
appid=
impid=*UDF Image Creator
winserialnum=0xbd061f8f
blocksize=2048
blocks=17898
usedblocks=9
freeblocks=0
behindblocks=17545
numfiles=2
numdirs=1
udfrev=1.02
udfwriterev=1.02
lastblock=608
integrity=closed
accesstype=readonly
softwriteprotect=no
hardwriteprotect=no
start=16, blocks=3, type=VRS
start=32, blocks=16, type=MVDS
start=64, blocks=1, type=LVID
start=128, blocks=16, type=RVDS
start=256, blocks=1, type=ANCHOR
start=320, blocks=9, type=PSPACE
start=352, blocks=1, type=ANCHOR

No issue, it is correct.

And now second:

$ ./udfinfo/udfinfo -b 2048 --startblock=12010 --lastblock=17898 image-ddrescue_11MB_2-sessions.udf
udfinfo: Warning: First Anchor Volume Descriptor Pointer not found, using second
udfinfo: Warning: Partition Space overlaps with other blocks
filename=image-ddrescue_11MB_2-sessions.udf
label=UDF-MulSess_2
uuid=2010112200000000
lvid=UDF-MulSess_2
vid=UDF-MulSess_2
vsid=
fsid=UDF-MulSess_2
fullvsid=20101122
owner=
organization=
contact=
appid=
impid=*UDF Image Creator
winserialnum=0xbd08075b
blocksize=2048
blocks=17898
usedblocks=17308
freeblocks=0
behindblocks=0
numfiles=4
numdirs=1
udfrev=1.02
udfwriterev=1.02
startblock=12010
lastblock=17898
integrity=closed
accesstype=readonly
softwriteprotect=no
hardwriteprotect=no
start=320, blocks=17308, type=PSPACE
start=12026, blocks=3, type=VRS
start=12042, blocks=16, type=MVDS
start=12074, blocks=1, type=LVID
start=12138, blocks=16, type=RVDS
start=17641, blocks=1, type=ANCHOR
start=17897, blocks=1, type=ANCHOR

Message udfinfo: Warning: Partition Space overlaps with other blocks can be ignored as on multisession disk data space may continue from one session to another or for space effectivity may overlap with other data (if it is properly created with holes).

But message udfinfo: Warning: First Anchor Volume Descriptor Pointer not found, using second is big warning as the first AVDP is entry point for UDF. Without first AVDP UDF filesystem is unusable. udfinfo is slightly smarter than other UDF implementations and can read AVDP from backup locations (and we are happy that there are two backups at the end of disk).

First AVDP must be at fixed block start+256. So at block 12266 for second session. Lets look what is stored at this block (via udfdump -S):

00012266    12266       TAG: descriptor 2, serial_num 1 at sector 256, crc length 496 bytes
                Anchor
                        Main    volume descriptor set at 12042 for 32768 bytes
                        Reserve volume descriptor set at 12138 for 32768 bytes

So AVDP is at block 12266, but has stored information in its data that is placed on block 256 (instead of 12266). 12266 != 256 meaning that content of block 12266 is broken and therefore ignored.

This seems to be a bug of software which generated second session of UDF filesystem.

On the other hand on block 17641 is stored backup of AVDP and it is correct (17641 = 17641):

00017641    17641       TAG: descriptor 2, serial_num 1 at sector 17641, crc length 496 bytes
                Anchor
                        Main    volume descriptor set at 12042 for 32768 bytes
                        Reserve volume descriptor set at 12138 for 32768 bytes

(so udfinfo used it)

@pali
Copy link

pali commented Feb 23, 2021

So I'm not surprised now that this udf multisession cd image cannot be detected, it is broken in a way that it is not valid udf image.

@pali
Copy link

pali commented Feb 23, 2021

Anyway, seems that correct way to call dd is with arguments conv=noerror,sync (noerror is not enough, that sync should add required zero bytes). Can you just recheck if dd conv=noerror,sync produce same output as dddrescue?

EDIT: seems that this is still not enough. According to https://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html additional iflag=fullblock is needed. So: dd conv=noerror,sync iflag=fullblock if=/dev/sr0 of=image.udf

@md-work
Copy link
Author

md-work commented Feb 23, 2021

Anyway, seems that correct way to call dd is with arguments conv=noerror,sync (noerror is not enough, that sync should add required zero bytes). Can you just recheck if dd conv=noerror,sync produce same output as dddrescue?

EDIT: seems that this is still not enough. According to https://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html additional iflag=fullblock is needed. So: dd conv=noerror,sync iflag=fullblock if=/dev/sr0 of=image.udf

Yes, it's identical with the ddrescue image.

 

[...]
This seems to be a bug of software which generated second session of UDF filesystem.

Bad thing...
Nevertheless, because most systems (including the Linux-Kernel and Windows - both tested) can handle this I guess this is a maybe common bug.

 

I finally found the physical disc I read the Test-Media-Mini image from. (see cdrdao image in udisks_udf_multisession_support.zip)
Here's an image read via ddrescue.
Test-Media-Mini_ddrescue.udf.gz

For this Test-Media-Mini a recent blkid.static compiled from master branch actually reports the correct label of the second session.
util-linux/util-linux@17d5b26 (compiled from this commit)
blkid-2.23.1 and blkid-2.23.2 still report the incorrect first session label.

@pali
Copy link

pali commented Feb 27, 2021

Yes, it's identical with the ddrescue image.

Ok, so we should always call dd with arguments conv=noerror,sync iflag=fullblock.

Test-Media-Mini_ddrescue.udf.gz

This image has same issue. And also has impid=*UDF Image Creator, so seems that this UDF Image Creator is broken...

@HeroesLoveToRoujiamo
Copy link

HeroesLoveToRoujiamo commented Mar 31, 2021

Regarding the multisession UDF, the kernel also needs to be fixed. This is my code submission, and it should not have been released yet:https://lore.kernel.org/lkml/20210114075741.30448-1-changlianzhi@uniontech.com/. Use with code updates related to the util-linux/util-linux#1161 issue.@pali

@pali
Copy link

pali commented Mar 31, 2021

I guess that above kernel patch should be backported also to stable kernel releases

@HeroesLoveToRoujiamo
Copy link

This is beyond my control, I actually think so too

@pali
Copy link

pali commented Mar 31, 2021

When sending such bugfix patch, just add Cc: stable@vger.kernel.org line into email body (commit message) and patch would be automatically backported (https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html). Also if you mention that patch is suitable for stable releases then maintainers would add that line...

@HeroesLoveToRoujiamo
Copy link

Thank you very much, I learned another knowledge point

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants