iSCSI support (not TYPE_DISK)

[behlendorf]

Ian Campbell reports:

I am using ZFS on Linux 0.6.0-rc6 with Ubuntu 10.04 x86 & amd64.

After getting the modules compiled I was able to everything installed fine but I seemed to have no run into a problem.

The problem is with the "new" unified target modules being added to the newer kernel to add native iSCSI support. More specifically the problem is related to the way the disks are being presented to the target software. When I attempt to add a device like /dev/zd0 or /dev/zvol/tank/fish. The target returns back to be saying the block device is not a "TYPE_DISK" and it won't add it.

Also when I view the disk with parted it has no vendor information, is it safe to assume the disk has no identifying information attached to such as the type of block device or pseudo vendor information?

Is there a way to resolve this issue? Can I add information to the vdev so it will be identified correctly?

If you need a better idea of what I am talking about I can give you any information you need.

Thank you for you time.

[ghost]

Pretty much this is how I went about setting up everything.

My platform is Ubuntu 10.04.3 x86 (also used x64) running as a VMware VM on my laptop (as test).

I downloaded the source and compiled as per the instructions on the ZFS on linux website.

Then I compiled the Linux iSCSI target core backports from linux-iscsi.org. I also compiled the lio-utils and targetcli (the management tools).

After I installed the deb packages of everything (iscsi target and ZFS) I created my zpool called (tank) and my zfs vol (fish). Because it was a test I just used the names from the website because it did not matter.

The command I use were the following;

parted /dev/sdb
mklabel gpt
quit

parted /dev/sdc
mklabel gpt
quit

zpool create tank mirror /dev/sdb /dev/sdc

zfs create tank/fish -V 18G

After that was done I dropped into the targetcli tool and tried to add a block device to the /backstores/iblock section. The targetcli emulates a file system, kind of reminds me of /proc or /sys. When I execute the command "create disk0 /dev/zd0" it returns and error to me saying the chosen device is not a valid "TYPE_DISK". I am not sure though if "TYPE_DISK" is something internal to the target or if it is a Linux thing.

The only way I could use ZFS with the target was to format the ZFS vol with something like ext4 and then create an image file with dd then use the file_io feature of the target. But the is not only complicated but completely undermines the entire point of using ZFS.

When I mentioned that the ZFS vols have no vendor information I was referring to what I see when I run "parted -l" and look at /dev/zd0. When compared to the VMware disks there is information about who made the disk or anything, not even faked information just to fill the space. I will add an output when I have a chance.

If you need anymore info let me know.

[rlaager]

Have you asked the developers on the iSCSI target side? Where can one download the code for that software?

[ghost]

I get the code from there git repo. git://risingtidesystems.com/

The only slightly annoying thing is you have to build several packages before you can build the targetcli tool.

But everything you need is there.

You need to build the tools in an similar order to this;

1. lio-utils
2. configshell
3. rtslib
4. targetcli

[ghost]

I have small update to announce based on my reported issue, I may have a source of the problem.

I believe the error "Not TYPE_DISK" is a problem with the iSCSI target drivers and may have nothing to do with ZFS.

The reason for the error I hypothoize is because ZFS is not listing it ZVOLs in /dev/disk which is most likely where the iSCSI target is look for them and that would sort of explain the error, because it is say that the ZVOL is not a type of device found is /dev/disk.

So unless something changes with the iSCSI target drivers before the "final" release with kernel v3.4 then ZFS just may have sit out on that one.

I will "try" to report the issue to the devs of the iSCSI target but I have not heard anything since before Christmas when I last attempted.

[behlendorf]

It should be relatively straight forward to update the zfs udev rules to make entries for zvols under /dev/disk/. Alternately, those entries are just symlinks to /dev/ devices, you could credate they manually for the purposes of a test.

[gclawes]

FWIW, the offending code is in rstlib, a support library for targetcli, specifically the function get_block_type() in utils.py.
http://www.risingtidesystems.com/git/?p=rtslib.git;a=blob;f=rtslib/utils.py;h=5c778f5592de26cda451919a8dc64bb0f9c9d855;hb=HEAD

get_block_type(path) is called when targetcli tries to add a block device as a backstore, and should return 0 for devices of TYPE_DISK. It doesn't for zvol device nodes, because the zvol devices' major numbers aren't listed in their huge list of acceptable numbers, and aren't caught by the fancy list-comprehension at the bottom because the there is no /sys/block/zdX/device/type entry for zvol device nodes.

EDIT: after adding the major number for /dev/zd0 to the list, targetcli was able to use it as a backstore.

def get_block_type(path):
    '''
    This function returns a block device's type.
    Example: 0 is TYPE_DISK
    If no match is found, None is returned.

    >>> from rtslib.utils import *
    >>> get_block_type("/dev/sda")
    0
    >>> get_block_type("/dev/sr0")
    5
    >>> get_block_type("/dev/scd0")
    5
    >>> get_block_type("/dev/nodevicehere") is None
    True

    @param path: path to the block device
    @type path: string
    @return: An int for the block device type, or None if not a block device.
    '''
    dev = os.path.realpath(path)
    # TODO: Make adding new majors on-the-fly possible, using some config file
    # for instance, maybe an additionnal list argument, or even a match all
    # mode for overrides ?

    # Make sure we are dealing with a block device
    (major, minor) = get_block_numbers(dev)
    if major is None:
        return None

    # Treat disk partitions as TYPE_DISK
    if is_disk_partition(path):
        return 0

    # These devices are disk type block devices, but might not report this
    # correctly in /sys/block/xxx/device/type, so use their major number.
    type_disk_known_majors = [1,    # RAM disk
                              8,    # SCSI disk devices
                              9,    # Metadisk RAID devices
                              13,   # 8-bit MFM/RLL/IDE controller
                              19,   # "Double" compressed disk
                              21,   # Acorn MFM hard drive interface
                              30,   # FIXME: Normally 'Philips LMS CM-205
                                    # CD-ROM' in the Linux devices list but
                                    # used by Cirtas devices.
                              35,   # Slow memory ramdisk
                              36,   # MCA ESDI hard disk
                              37,   # Zorro II ramdisk
                              43,   # Network block devices
                              44,   # Flash Translation Layer (FTL) filesystems
                              45,   # Parallel port IDE disk devices
                              47,   # Parallel port ATAPI disk devices
                              48,   # Mylex DAC960 PCI RAID controller
                              48,   # Mylex DAC960 PCI RAID controller
                              49,   # Mylex DAC960 PCI RAID controller
                              50,   # Mylex DAC960 PCI RAID controller
                              51,   # Mylex DAC960 PCI RAID controller
                              52,   # Mylex DAC960 PCI RAID controller
                              53,   # Mylex DAC960 PCI RAID controller
                              54,   # Mylex DAC960 PCI RAID controller
                              55,   # Mylex DAC960 PCI RAID controller
                              58,   # Reserved for logical volume manager
                              59,   # Generic PDA filesystem device
                              60,   # LOCAL/EXPERIMENTAL USE
                              61,   # LOCAL/EXPERIMENTAL USE
                              62,   # LOCAL/EXPERIMENTAL USE
                              63,   # LOCAL/EXPERIMENTAL USE
                              64,   # Scramdisk/DriveCrypt encrypted devices
                              65,   # SCSI disk devices (16-31)
                              66,   # SCSI disk devices (32-47)
                              67,   # SCSI disk devices (48-63)
                              68,   # SCSI disk devices (64-79)
                              69,   # SCSI disk devices (80-95)
                              70,   # SCSI disk devices (96-111)
                              71,   # SCSI disk devices (112-127)
                              72,   # Compaq Intelligent Drive Array
                              73,   # Compaq Intelligent Drive Array
                              74,   # Compaq Intelligent Drive Array
                              75,   # Compaq Intelligent Drive Array
                              76,   # Compaq Intelligent Drive Array
                              77,   # Compaq Intelligent Drive Array
                              78,   # Compaq Intelligent Drive Array
                              79,   # Compaq Intelligent Drive Array
                              80,   # I2O hard disk
                              80,   # I2O hard disk
                              81,   # I2O hard disk
                              82,   # I2O hard disk
                              83,   # I2O hard disk
                              84,   # I2O hard disk
                              85,   # I2O hard disk
                              86,   # I2O hard disk
                              87,   # I2O hard disk
                              93,   # NAND Flash Translation Layer filesystem
                              94,   # IBM S/390 DASD block storage
                              96,   # Inverse NAND Flash Translation Layer
                              98,   # User-mode virtual block device
                              99,   # JavaStation flash disk
                              101,  # AMI HyperDisk RAID controller
                              102,  # Compressed block device
                              104,  # Compaq Next Generation Drive Array
                              105,  # Compaq Next Generation Drive Array
                              106,  # Compaq Next Generation Drive Array
                              107,  # Compaq Next Generation Drive Array
                              108,  # Compaq Next Generation Drive Array
                              109,  # Compaq Next Generation Drive Array
                              110,  # Compaq Next Generation Drive Array
                              111,  # Compaq Next Generation Drive Array
                              112,  # IBM iSeries virtual disk
                              114,  # IDE BIOS powered software RAID interfaces
                              115,  # NetWare (NWFS) Devices (0-255)
                              117,  # Enterprise Volume Management System
                              120,  # LOCAL/EXPERIMENTAL USE
                              121,  # LOCAL/EXPERIMENTAL USE
                              122,  # LOCAL/EXPERIMENTAL USE
                              123,  # LOCAL/EXPERIMENTAL USE
                              124,  # LOCAL/EXPERIMENTAL USE
                              125,  # LOCAL/EXPERIMENTAL USE
                              126,  # LOCAL/EXPERIMENTAL USE
                              127,  # LOCAL/EXPERIMENTAL USE
                              128,  # SCSI disk devices (128-143)
                              129,  # SCSI disk devices (144-159)
                              130,  # SCSI disk devices (160-175)
                              131,  # SCSI disk devices (176-191)
                              132,  # SCSI disk devices (192-207)
                              133,  # SCSI disk devices (208-223)
                              134,  # SCSI disk devices (224-239)
                              135,  # SCSI disk devices (240-255)
                              136,  # Mylex DAC960 PCI RAID controller
                              137,  # Mylex DAC960 PCI RAID controller
                              138,  # Mylex DAC960 PCI RAID controller
                              139,  # Mylex DAC960 PCI RAID controller
                              140,  # Mylex DAC960 PCI RAID controller
                              141,  # Mylex DAC960 PCI RAID controller
                              142,  # Mylex DAC960 PCI RAID controller
                              143,  # Mylex DAC960 PCI RAID controller
                              144,  # Non-device (e.g. NFS) mounts
                              145,  # Non-device (e.g. NFS) mounts
                              146,  # Non-device (e.g. NFS) mounts
                              147,  # DRBD device
                              152,  # EtherDrive Block Devices
                              153,  # Enhanced Metadisk RAID storage units
                              160,  # Carmel 8-port SATA Disks
                              161,  # Carmel 8-port SATA Disks
                              199,  # Veritas volume manager (VxVM) volumes
                              201,  # Veritas VxVM dynamic multipathing driver
                              240,  # LOCAL/EXPERIMENTAL USE
                              241,  # LOCAL/EXPERIMENTAL USE
                              242,  # LOCAL/EXPERIMENTAL USE
                              243,  # LOCAL/EXPERIMENTAL USE
                              244,  # LOCAL/EXPERIMENTAL USE
                              245,  # LOCAL/EXPERIMENTAL USE
                              246,  # LOCAL/EXPERIMENTAL USE
                              247,  # LOCAL/EXPERIMENTAL USE
                              248,  # LOCAL/EXPERIMENTAL USE
                              249,  # LOCAL/EXPERIMENTAL USE
                              250,  # LOCAL/EXPERIMENTAL USE
                              251,  # LOCAL/EXPERIMENTAL USE
                              252,  # LOCAL/EXPERIMENTAL USE
                              253,  # LOCAL/EXPERIMENTAL USE
                              254   # LOCAL/EXPERIMENTAL USE
                             ]
    if major in type_disk_known_majors:
        return 0

    # Same for LVM LVs, but as we cannot use major here
    # (it varies accross distros), use the realpath to check
    if os.path.dirname(dev) == "/dev/mapper":
        return 0

    # list of (major, minor, type) tuples
    blocks = [(fread("%s/dev" % fdev).strip().split(':')[0],
        fread("%s/dev" % fdev).strip().split(':')[1],
        fread("%s/device/type" % fdev).strip())
        for fdev in glob.glob("/sys/block/*")
        if os.path.isfile("%s/device/type" % fdev)]

    for block in blocks:
        if int(block[0]) == major and int(block[1]) == minor:
            return int(block[2])

    return None

[behlendorf]

Great news, thanks for posting the root cause. It sounds like there are two possible solutions here:

1. Push your patch to the upstream developers to add the zvol major number to their white list.

2. Request that zvols use a major number in their existing list with the zvol_major module option.

/sbin/insmod zfs.ko zvol_major=240

[gclawes]

Is there any particular reason zvols use the current major number of 230?

[behlendorf]

The choice of 230 was some what arbitrary but it was selected to avoid the reserved/local major numbers and it was unused as far as I could tell. We needed to ensure it didn't conflict with any existing known drivers, which almost be definition means it shouldn't appear on the above white list.

[gclawes]

fix submitted to targetcli mailing list, CC'd to zfs-devel@zfsonlinux.org

[chtitux]

Just FYI, the patch has been added there :
http://www.risingtidesystems.com/git/?p=rtslib.git;a=commitdiff;h=655e87fcf85040d51f90d3fea4b550f33388319f

The patch is not in rtslib 2.1.