New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot force 4k #72

Closed
priyadarshan opened this Issue Sep 4, 2018 · 27 comments

Comments

Projects
None yet
3 participants
@priyadarshan

priyadarshan commented Sep 4, 2018

Installing OmniOS on NVMe Samsung 960 Pro, having set Force 4K Sectors to YES, OmniOS installer signals:

Cannot unload module: sd
Will be unloaded upon reboot.
Forcing update of sd.conf.
failed to read label from /dev/dsk/c4t0025385971B16535d0s0

screenshot 2018-09-04 at 09 47 57

Checking post-install, ashift is indeed 9.

Unfortunately I realised this after doing many installs.

The error message is flashing too quickly to recognise and it did not come to me to double check until now.

By the way, many SSD, including Samsung 950, 960, 970 are 8K drives, it seems they should have ashift=13.

It would be nice to have option to force either 4K or 8K.

Thank you.

@citrus-it

This comment has been minimized.

Member

citrus-it commented Sep 4, 2018

Can you post the contents of /kernel/drv/sd.conf on the installed system?
My guess would be that we need to do this a different way for NVMe.

@citrus-it

This comment has been minimized.

Member

citrus-it commented Sep 4, 2018

Can you try dropping to the shell in the installer before starting installation, and typing:

echo "min-phys-block-size=4096;" >> /kernel/drv/nvme.conf
update_drv -f nvme
exit

Then do the installation and see if you get ashift=12?
(or try 8192 for ashift=13)

@priyadarshan

This comment has been minimized.

priyadarshan commented Sep 4, 2018

Can you post the contents of /kernel/drv/sd.conf on the installed system?

root@omnis:~# cat /kernel/drv/sd.conf
#
# CDDL HEADER START
#
# The contents of this file are subject to the terms of the
# Common Development and Distribution License (the "License").
# You may not use this file except in compliance with the License.
#
# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
# or http://www.opensolaris.org/os/licensing.
# See the License for the specific language governing permissions
# and limitations under the License.
#
# When distributing Covered Code, include this CDDL HEADER in each
# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
# If applicable, add the following below this CDDL HEADER, with the
# fields enclosed by brackets "[]" replaced with your own identifying
# information: Portions Copyright [yyyy] [name of copyright owner]
#
# CDDL HEADER END
#

#
# Copyright (c) 1991, 2010, Oracle and/or its affiliates. All rights reserved.
#

name="sd" class="scsi" target=0 lun=0;
name="sd" class="scsi" target=1 lun=0;
name="sd" class="scsi" target=2 lun=0;
name="sd" class="scsi" target=3 lun=0;
name="sd" class="scsi" target=4 lun=0;
name="sd" class="scsi" target=5 lun=0;
name="sd" class="scsi" target=6 lun=0;
name="sd" class="scsi" target=7 lun=0;
name="sd" class="scsi" target=8 lun=0;
name="sd" class="scsi" target=9 lun=0;
name="sd" class="scsi" target=10 lun=0;
name="sd" class="scsi" target=11 lun=0;
name="sd" class="scsi" target=12 lun=0;
name="sd" class="scsi" target=13 lun=0;
name="sd" class="scsi" target=14 lun=0;
name="sd" class="scsi" target=15 lun=0;

#
# The following stub node is needed for pathological bottom-up
# devid resolution on a self-identifying transport.
#
name="sd" class="scsi-self-identifying";

#
# Associate the driver with devid resolution.
#
ddi-devid-registrant=1;
@priyadarshan

This comment has been minimized.

priyadarshan commented Sep 4, 2018

Can you try dropping to the shell in the installer before starting installation, and typing:

It fails with an error that flashes through the screen. I am now taking a video so I can read it and post it here.

@priyadarshan

This comment has been minimized.

priyadarshan commented Sep 4, 2018

I tried twice, just in case, but ashift is still 9:

root@omniosce:~# zdb
rpool:
    version: 5000
    name: 'rpool'
    state: 0
    txg: 131
    pool_guid: 1357762110761258899
    hostid: 146670196
    hostname: 'omniosce'
    com.delphix:has_per_vdev_zaps
    vdev_children: 1
    vdev_tree:
        type: 'root'
        id: 0
        guid: 1357762110761258899
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 17958557208562143769
            path: '/dev/dsk/c4t0025385971B16535d0s1'
            devid: 'id1,kdev@w0025385971b16535/b'
            phys_path: '/pci@37,0/pci1022,1453@1,1/pci144d,a801@0/blkdev@w0025385971B16535,0:b'
            whole_disk: 1
            metaslab_array: 38
            metaslab_shift: 32
            ashift: 9
            asize: 511828295680
            is_log: 0
            create_txg: 4
            com.delphix:vdev_zap_leaf: 36
            com.delphix:vdev_zap_top: 37
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data

This is the screenshot, taken after the suggested edit. I did cat /kernel/drv/nvme.conf to show file content after edit. I am not sure if that message is good or not.

00004014

@citrus-it

This comment has been minimized.

Member

citrus-it commented Sep 5, 2018

Could you give https://downloads.omniosce.org/media/bloody/omniosce-bloody-20180905.iso a spin please?
This one allows selection of 4K or 8K as well as leaving it to whatever the disk advertises.

@priyadarshan

This comment has been minimized.

priyadarshan commented Sep 5, 2018

Thank you so much.

Do you happen to have a usb-dd version? I am struggling to "burn" that iso to thumbdrive, using either unetbooting, rufus, iso2usb or Etcher.

@citrus-it

This comment has been minimized.

@priyadarshan

This comment has been minimized.

priyadarshan commented Sep 5, 2018

Thank you, usb-dd version imaged with no issues.

I installed it twice, to test both 4K and 8K.

priyadarshan@omniosce:~$ uname -a
SunOS omniosce 5.11 omnios-master-36eb49126f i86pc i386 i86pc

First as 4K, to a Samsung 960 Pro 512GB:

priyadarshan@omniosce:~$ zdb
rpool:
    version: 5000
    name: 'rpool'
    state: 0
    txg: 106
    pool_guid: 2254942589280409017
    hostid: 146670196
    hostname: ''
    com.delphix:has_per_vdev_zaps
    vdev_children: 1
    vdev_tree:
        type: 'root'
        id: 0
        guid: 2254942589280409017
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 14092311627064017026
            path: '/dev/dsk/c4t0025385971B16535d0s1'
            devid: 'id1,kdev@w0025385971b16535/b'
            phys_path: '/pci@38,0/pci1022,1453@1,1/pci144d,a801@0/blkdev@w0025385971B16535,0:b'
            whole_disk: 1
            metaslab_array: 38
            metaslab_shift: 32
            ashift: 12
            asize: 511828295680
            is_log: 0
            create_txg: 4
            com.delphix:vdev_zap_leaf: 36
            com.delphix:vdev_zap_top: 37
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data

Then as 8K, to a Samsung 960 Pro 1TB:

store@omniosce8k:~$ zdb
rpool:
    version: 5000
    name: 'rpool'
    state: 0
    txg: 128
    pool_guid: 397576901342805206
    hostid: 146670196
    hostname: 'omniosce8k'
    com.delphix:has_per_vdev_zaps
    vdev_children: 1
    vdev_tree:
        type: 'root'
        id: 0
        guid: 397576901342805206
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 17656612001213952747
            path: '/dev/dsk/c5t0025385181B144D5d0s1'
            devid: 'id1,kdev@w0025385181b144d5/b'
            phys_path: '/pci@38,0/pci1022,1453@1,2/pci144d,a801@0/blkdev@w0025385181B144D5,0:b'
            whole_disk: 1
            metaslab_array: 38
            metaslab_shift: 33
            ashift: 13
            asize: 1023927648256
            is_log: 0
            create_txg: 4
            com.delphix:vdev_zap_leaf: 36
            com.delphix:vdev_zap_top: 37
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data

Ashift values now are set-up as ashift=12 and ashift=13 respectively.

This is excellent, thank you so much!

@priyadarshan

This comment has been minimized.

priyadarshan commented Sep 5, 2018

One small thing I noticed, in the first case during install I let hostname keep default value "omniosce", and zdb reports empty field, hostname: '' instead of hostname: 'omniosce'.

Second install, I specified hostname "omniosce8k", and zdb reports hostname: 'omniosce8k'.

@priyadarshan

This comment has been minimized.

priyadarshan commented Sep 5, 2018

Is there a way to use this on OmnisOS r151026 as well, perhaps via shell at install time?

@citrus-it

This comment has been minimized.

Member

citrus-it commented Sep 6, 2018

Not just yet, unfortunately the commit that allows for overriding the ashift at the ZFS level is not in r151026 (omniosorg/illumos-omnios@93a1902)
I will get it backported and then I can update the installer for r151026 too; it will take a couple of days.

@priyadarshan

This comment has been minimized.

priyadarshan commented Sep 6, 2018

Thank you for considering it.
I needed to reinstall some servers over the week-end, but I can redo them again later on.
I am very grateful for the work, very much appreciated!

@oetiker

This comment has been minimized.

Member

oetiker commented Sep 6, 2018

@priyadarshan https://omniosce.org/patron.html and https://omniosce.org/invoice.html are also excellent tools for expressing that OmniOS CE is providing essential services for your business.

@priyadarshan

This comment has been minimized.

priyadarshan commented Sep 6, 2018

Thank you, I knew about those. I do this as volunteer work for a non-profit, with basically no budget. But I will tell the board!

@citrus-it

This comment has been minimized.

Member

citrus-it commented Sep 11, 2018

Please try these images:

These include the update for properly setting ashift at installation time, and also an upcoming fix for the L1TF/Foreshadow CVE which we are currently soak testing and plan to release in the next week or two.

@priyadarshan

This comment has been minimized.

priyadarshan commented Sep 12, 2018

Thank you for the images. I have download the usb-dd one, since I had issues with iso in the past.

After booting and pressing Enter to skip 9 sec. delay, this message is printed after some other messages:

Configuring devices.
Applying initial boot settings...

Scanning for media...
NOTICE: mount: not a UFS magic number (0x0)
NOTICE: mount: not a UFS magic number (0x0)
NOTICE: mount: not a UFS magic number (0x0)
NOTICE: mount: not a UFS magic number (0x0)

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!! Could not mount installation media !!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

This is a almost-bare machine, no adapters, only 2 NVMe drives.

I tested with two different usb thumbdrives, both working (as of yesterday) as OmniOS install media.

Sha256 was verified.

@citrus-it

This comment has been minimized.

Member

citrus-it commented Sep 12, 2018

Could you drop to a shell and run /kayak/bin/mount_media -v r151026r? That should give some more output on what is going wrong.

@priyadarshan

This comment has been minimized.

priyadarshan commented Sep 12, 2018

Thanks, I dropped to shell, that command printed:

bash-4.4# /kayak/bin/mount_media -v r151026r
NOTICE: mount: not a UFS magic number (0x0)
NOTICE: mount: not a UFS magic number (0x0)
NOTICE: mount: not a UFS magic number (0x0)
NOTICE: mount: not a UFS magic number (0x0)
bash-4.4#
@citrus-it

This comment has been minimized.

Member

citrus-it commented Sep 12, 2018

Can you try https://downloads.omniosce.org/media/r151026/omniosce-r151026r-pre2.usb-dd ?
I just booted this on a USB flash drive in an HP DL380 and it was able to mount the installation media.
If nothing else, the mount_media command should provide more output with this version.

@priyadarshan

This comment has been minimized.

priyadarshan commented Sep 12, 2018

Thank you, doing it right now. I was surprised with that error as the same USB boots fine with both r151026 and r151027.

@priyadarshan

This comment has been minimized.

priyadarshan commented Sep 12, 2018

Thank you, this time it worked smoothly. I have installed it, and I can confirm ashift = 13 is set:

root@8k:~# zdb | grep ashift
            ashift: 13

I will also try to install as 4k to different NVMe, and report back.

@priyadarshan

This comment has been minimized.

priyadarshan commented Sep 12, 2018

Setting 4k works too:

root@4k:~# zdb | grep ashift
            ashift: 12

Thank you so much for your time into this. I know how busy you are!

Thank you.

@citrus-it

This comment has been minimized.

Member

citrus-it commented Sep 13, 2018

I'll close this now - when we release updated media for r151026 in the next week or two it will include this fix and you should be ok to use the pre2 media if you want, it contains the same packages that we will be releasing.

@citrus-it citrus-it closed this Sep 13, 2018

@priyadarshan

This comment has been minimized.

priyadarshan commented Sep 14, 2018

Thank you again.

@priyadarshan

This comment has been minimized.

priyadarshan commented Sep 16, 2018

Just a note:

rpool as created by last May r151026 installer has atime set to off.

root@chstore:~# uname -a
SunOS chstore 5.11 omnios-r151026-673c59f55d i86pc i386 i86pc
root@chstore:~# zfs get atime rpool
NAME   PROPERTY  VALUE  SOURCE
rpool  atime     off    local

rpool as created by September r151026 installer has atime set to on.

root@argo:~# uname -a
SunOS argo 5.11 omnios-r151026-d258f672aa i86pc i386 i86pc
root@argo:~# zfs get atime rpool
NAME   PROPERTY  VALUE  SOURCE
rpool  atime     on     default
@priyadarshan

This comment has been minimized.

priyadarshan commented Sep 16, 2018

Wait, the last machine has napp-it installed as well. Doing zpool history shows atime was set at later time:

2018-09-09.15:18:35 zpool create -fB rpool c4t3d0
2018-09-09.15:19:28 zfs set compression=on rpool
2018-09-09.15:19:28 zfs create rpool/ROOT
2018-09-09.15:19:28 zfs set canmount=off rpool/ROOT
2018-09-09.15:19:28 zfs set mountpoint=legacy rpool/ROOT
2018-09-09.15:19:54 zfs receive -u rpool/ROOT/omnios
2018-09-09.15:19:54 zfs set canmount=noauto rpool/ROOT/omnios
2018-09-09.15:19:54 zfs set mountpoint=legacy rpool/ROOT/omnios
2018-09-09.15:19:54 zfs set org.opensolaris.libbe:uuid=efa8d944-9a59-41e4-c1a5-b6d165b1aa0c rpool/ROOT/omnios
2018-09-09.15:19:54 zfs set org.opensolaris.libbe:policy=static rpool/ROOT/omnios
2018-09-09.15:19:54 zfs create -V 4G -b 4096 -o logbias=throughput -o sync=always -o primarycache=metadata -o secondarycache=none rpool/swap
2018-09-09.15:19:54 zfs create -V 4G -b 4096 -o logbias=throughput -o sync=always -o primarycache=metadata -o secondarycache=none rpool/dump
2018-09-09.15:19:54 zfs destroy rpool/ROOT/omnios@kayak
2018-09-09.15:19:54 zpool set bootfs=rpool/ROOT/omnios rpool
2018-09-09.15:45:18 zfs set atime=off rpool
2018-09-09.15:45:21 zfs set compress=lz4 rpool
2018-09-09.15:47:21 zfs set atime=off rpool
2018-09-09.15:48:15 zpool status

Sory for the noise 😥

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment