ndctl create-namespace over NVDIMM-N fails #78

sujithshankar · 2019-01-09T07:31:00Z

Description:
I have a freshly erased and factory reset (through BIOS settings) 2x 16GB NVDIMM-Ns.
Boot to kernel-4.20 installed over RHEL 7.6

Check the presence of nmems -
#ls /dev/nmem*
/dev/nmem0 /dev/nmem1

Check the presence of regions -
#ndctl list -R
[
{
"dev":"region1",
"size":17179869184,
"available_size":17179869184,
"max_available_extent":17179869184,
"type":"pmem",
"iset_id":6272005994369602076,
"persistence_domain":"unknown"
},
{
"dev":"region0",
"size":17179869184,
"available_size":17179869184,
"max_available_extent":17179869184,
"type":"pmem",
"iset_id":6274820741989552668,
"persistence_domain":"unknown"
}
]

Try creating namespaces:
#ndctl create-namespace -r region0 -f -vvvv
libndctl: __sysfs_device_parse: base: /sys/class/nd dev: ndctl
libndctl: __sysfs_device_parse: ndctl0: processed
libndctl: __sysfs_device_parse: base: /sys/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0012:00/ndbus0 dev: region
libndctl: to_module: alias: nd:t2 module: libnvdimm
libndctl: region_set_type: region0: iset-0x5714a4382b8a521c added
libndctl: __sysfs_read_attr: failed to open /sys/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0012:00/ndbus0/region0/persistence_domain: No such file or directory
libndctl: __sysfs_device_parse: region0: processed
libndctl: to_module: alias: nd:t2 module: libnvdimm
libndctl: region_set_type: region1: iset-0x570aa438ab85521c added
libndctl: __sysfs_read_attr: failed to open /sys/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0012:00/ndbus0/region1/persistence_domain: No such file or directory
libndctl: __sysfs_device_parse: region1: processed
libndctl: __sysfs_device_parse: base: /sys/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0012:00/ndbus0/region0 dev: pfn0.
libndctl: to_module: alias: nd:t5 module: nd_pmem
libndctl: __sysfs_read_attr: failed to read /sys/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0012:00/ndbus0/region0/pfn0.0/resource: No such device or address
libndctl: __sysfs_read_attr: failed to read /sys/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0012:00/ndbus0/region0/pfn0.0/size: No such device or address
libndctl: __sysfs_device_parse: pfn0.0: processed
libndctl: __sysfs_device_parse: base: /sys/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0012:00/ndbus0/region0 dev: dax0.
libndctl: to_module: alias: nd:t7 module: dax_pmem
libndctl: __sysfs_read_attr: failed to read /sys/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0012:00/ndbus0/region0/dax0.0/resource: No such device or address
libndctl: __sysfs_read_attr: failed to read /sys/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0012:00/ndbus0/region0/dax0.0/size: No such device or address
libndctl: __sysfs_device_parse: dax0.0: processed
libndctl: __sysfs_device_parse: base: /sys/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0012:00/ndbus0/region0 dev: namespace0.
libndctl: __sysfs_read_attr: failed to read /sys/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0012:00/ndbus0/region0/namespace0.0/resource: No such device or address
libndctl: parse_lbasize_supported: namespace0.0: 512 4096
libndctl: to_module: alias: nd:t5 module: nd_pmem
libndctl: __sysfs_device_parse: namespace0.0: processed
libndctl: __sysfs_device_parse: base: /sys/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0012:00/ndbus0/region0 dev: btt0.
libndctl: to_module: alias: nd:t5 module: nd_pmem
libndctl: parse_lbasize_supported: btt0.0: 512 520 528 4096 4104 4160 4224
libndctl: __sysfs_read_attr: failed to read /sys/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0012:00/ndbus0/region0/btt0.0/size: No such device or address
libndctl: __sysfs_device_parse: btt0.0: processed
libndctl: write_attr: failed to write 27283b48-a90f-49b4-ab93-5db7427cd7c8 to /sys/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0012:00/ndbus0/region0/namespace0.0/uuid: Device or resource busy
setup_namespace:347: namespace0.0: set_uuid failed: Device or resource busy
failed to create namespace: Device or resource busy

Version:
kernel-4.20
ndctl v62

Could you let me know what might be causing the problem?

djbw · 2019-01-09T08:34:54Z

Hmm, could you also add the output of dmesg?

sujithshankar · 2019-01-09T08:51:31Z

Attaching the dmesg collected after the create-namspace failure.
Let me know if you need anything else.
dmesg.txt

djbw · 2019-01-09T09:30:00Z

Unfortunately nothing interesting in that dmesg. Can you do the following:

ndctl disable-region all
modprobe -r nfit
echo "module libnvdimm +p" > /sys/kernel/debug/dynamic_debug/control
modprobe nfit

...and then send the result of dmesg after attempting to create a namespace. That should capture detailed debug information.

sujithshankar · 2019-01-09T10:26:07Z

Had some trouble to unloading nfit, so used boot param -
debug verbose ignore_loglevel libnvdimm.dyndbg=+fp nfit.dyndbg=+fp
Hope this should be fine for this analysis..

dmesg_filtered_namespace_fail.txt
dmesg.txt

sujithshankar · 2019-01-09T14:22:32Z

Not sure if this helps..
I added a printk in kernel namespace_update_uuid() function and looks like it is returning here:
if (list_empty(&nd_mapping->labels))
return -EBUSY;

btomar-pmem · 2019-01-09T15:02:17Z

When you sanitized nvdimm's from BIOS, did you end up with default namespaces, something like "namespace0.0", which were then deleted by you to free up space in the region ?

May be your nvdimm's don't support labels. Which h/w platform are you using ?

sujithshankar · 2019-01-09T15:18:27Z

No. After sanitizing the NVDIMMs, no namespaces come automatically after boot. I can see only the regions as mentioned in Description.

It is a Dell PowerEdge 14G server, it does support NVDIMM-N.
I am using it with latest NVDIMM firmware web posted.

sujithshankar · 2019-01-09T16:04:15Z

Point to note - the change introduced by latest nvdimm fw-
The FW package provides the ability to create and manage label typed block data as defined in the JEDEC BAEBI 2.1 specification. This spec calls for an additional scratchpad region on the DIMM to be able to manage/configure labels or namespaces.

With the previous fw, there were no labels implemented by the platform and pmem/namespaces were auto enumerated.

djbw · 2019-01-10T01:03:42Z

Hi, I'm not seeing the namespace creation attempt in the log, however I do see a potential BIOS problem. The BIOS reports that ARS is complete to an "ARS Status" request, but any attempt to perform "ARS Start" returns "Busy". If ARS is busy then "ARS Status" should say so and Linux will exponentially back-off on the polling interval otherwise Linux will retry every second until ARS start succeeds.

Back to the namespace creation problem...

Can you retry the capture with just "libnvdimm.dyndbg=+fp", i.e. drop the "nfit.dyndbg=+fp".

You can mark in dmesg where the namespace creation attempt starts by doing something like:

echo "===== creating namespace ========" > /dev/kmsg

sujithshankar · 2019-01-10T16:22:36Z

I had to give up the original setup. Got a new setup having the same repro.
After these settings, tried create-namespace twice.
dmesg.txt

sujithshankar · 2019-01-10T16:23:33Z

Always the very first create-namespace try after reboot fails with 'inappropriate ioctl', further tries fails with 'Device busy'

djbw · 2019-01-11T00:43:07Z

I think this is a case where the DIMM does not have an index-block initialized. When ndctl attempts to initialize the index-block it uses a routine that assumes a 128K label area. For a 1K label area it needs a fix that was already made to the kernel, but was never duplicated in ndctl.

I pushed out a branch:

https://github.com/pmem/ndctl/tree/issues/78
7698a5a

...with a proposed fix. Let me know if that fixes it for you.

You might need to perform the following to get the labels initialized:

ndctl disable-region all
ndctl init-labels all
ndctl enable-region all

sujithshankar · 2019-01-11T04:36:44Z

No luck.
I applied this patch over ndctl v62, then make uninstall; make; make install;

#ndctl disable-region all
disabled 2 regions

#ndctl init-labels all
nmem1: error: labels already initialized
nmem3: error: labels already initialized
nmem5: error: labels already initialized
nmem10: error: labels already initialized
nmem7: error: labels already initialized
nmem9: error: labels already initialized
nmem0: error: labels already initialized
nmem2: error: labels already initialized
nmem4: error: labels already initialized
nmem6: error: labels already initialized
nmem11: error: labels already initialized
nmem8: error: labels already initialized
initialized 0 nmem

#ndctl enable-region all
enabled 2 regions

#ndctl create-namespace -r "region0" -f
failed to create namespace: Inappropriate ioctl for device

#ndctl create-namespace -r "region0" -f
failed to create namespace: Device or resource busy

djbw · 2019-01-11T04:57:11Z

ah, sorry add the --force option to init-labels:

ndctl init-labels all -f

sujithshankar · 2019-01-11T05:00:09Z

Actually I too missed out mentioning it here:
#ndctl init-labels all -f
initialized 0 nmem

Logs indicate ioctl failure.
ndctl init-labels fail.txt

djbw · 2019-01-11T05:31:52Z

Thanks for that. This shows the ioctl failure is specific to the label write path. Can you send the output of the following:

cd /sys/bus/nd/devices/nmem0/
cat commands
cat nfit/dsm_mask
cat nfit/family

sujithshankar · 2019-01-11T05:53:26Z

#cat commands
get_size get_data set_data cmd_call

#cat nfit/dsm_mask
0x4003c3f

#cat nfit/family
3

djbw · 2019-01-11T05:57:54Z

I think I might see the kernel bug. Can you try the following setting the override_dsm_mask

ndctl disable-region all
modprobe -r nfit
modprobe nfit override_dsm_mask=0x4003c7f

The kernel is improperly assuming the DIMM write commands are not supported.

sujithshankar · 2019-01-11T06:31:38Z

I don't know if this is a defect or not but I cannot unload the nfit nor the libnvdimm. Both seems to have a cyclic dependency on each other.
#lsmod | grep -i nfit
nfit 65536 2
libnvdimm 176128 1 nfit

Anyways, I booted with this kernel param "nfit.override_dsm_mask=0x4003c7f"

#cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-4.20.1 root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap libnvdimm.dyndbg=+fp debug verbose ignore_loglevel nfit.override_dsm_mask=0x4003c7f

Same failure is seen.

#ndctl create-namespace -r "region0" -f
failed to create namespace: Inappropriate ioctl for device

#ndctl create-namespace -r "region0" -f
failed to create namespace: Device or resource busy

djbw · 2019-01-11T06:44:09Z

Well, at least you're finding more kernel bugs and making it clear that I need to get one of these platforms for a regression environment. It turns out that override_dsm_mask does not force enable the DSMs.

You'll need to recompile the driver, and I think this one line change will address the problem:

diff --git a/drivers/acpi/nfit/core.c b/drivers/acpi/nfit/core.c
index 4ae953a5aebf..e0c42d46da86 100644
--- a/drivers/acpi/nfit/core.c
+++ b/drivers/acpi/nfit/core.c
@@ -470,7 +470,7 @@ int acpi_nfit_ctl(struct nvdimm_bus_descriptor *nd_desc, struct nvdimm *nvdimm,
        if (!desc || (cmd && (desc->out_num + desc->in_num == 0)))
                return -ENOTTY;
 
-       if (!test_bit(cmd, &cmd_mask) || !test_bit(func, &dsm_mask))
+       if (!test_bit(cmd, &cmd_mask) && !test_bit(func, &dsm_mask))
                return -ENOTTY;
 
        in_obj.type = ACPI_TYPE_PACKAGE;

sujithshankar · 2019-01-11T10:09:52Z

Verified in a couple of different Dell servers, where the issue was originally seen, that with the above kernel fix over v4.20.1, create-namespace works fine using ndctl v62.
Could you please help getting this fix upstream?
Thanks!

JulietDeltaGolf · 2019-01-14T17:47:32Z

Had the same problem on an ubuntu 18.04.1 and a Dell PowerEdge R840.
Applied the patch on the HWE kernel and the NVDIMMs works fine now.

Many Thanks !

The _DSM function number validation only happens to succeed when the generic Linux command number translation corresponds with a DSM-family-specific function number. This breaks NVDIMM-N implementations that correctly implement _LSR, _LSW, and _LSI, but do not happen to publish support for DSM function numbers 4, 5, and 6. Recall that the support for _LS{I,R,W} family of methods results in the DIMM being marked as supporting those command numbers at acpi_nfit_register_dimms() time. The DSM function mask is only used for ND_CMD_CALL support of non-NVDIMM_FAMILY_INTEL devices. Fixes: 31eca76 ("nfit, libnvdimm: limited/whitelisted dimm command...") Cc: <stable@vger.kernel.org> Link: pmem/ndctl#78 Cc: Jeff Moyer <jmoyer@redhat.com> Reported-by: Sujith Pandel <sujith_pandel@dell.com> Tested-by: Sujith Pandel <sujith_pandel@dell.com> Reviewed-by: Vishal Verma <vishal.l.verma@intel.com> Signed-off-by: Dan Williams <dan.j.williams@intel.com>

Quoting from Linux kernel commit 9e694d9c18dd "libnvdimm, label: change nvdimm_num_label_slots per UEFI 2.7": sizeof_namespace_index() fails when NVDIMM devices have the minimum 1024 bytes label storage area. nvdimm_num_label_slots() returns 3 slots while the area is only big enough for 2 slots. Change nvdimm_num_label_slots() to calculate a number of label slots according to UEFI 2.7 spec. Without this fix attempts to initialize labels on a small (1K) label area results in the following: libndctl: sizeof_namespace_index: nmem2: label area (1024) too small to host (128 byte) labels libndctl: sizeof_namespace_index: nmem2: label area (1024) too small to host (256 byte) labels Based on an original patch by Toshi Kani Fixes: bdaec95 ("ndctl: introduce ndctl_dimm_{validate_labels,init_labels}") Reported-by: Sujith Pandel <sujith_pandel@dell.com> Link: #78 Signed-off-by: Dan Williams <dan.j.williams@intel.com> Signed-off-by: Vishal Verma <vishal.l.verma@intel.com>

The _DSM function number validation only happens to succeed when the generic Linux command number translation corresponds with a DSM-family-specific function number. This breaks NVDIMM-N implementations that correctly implement _LSR, _LSW, and _LSI, but do not happen to publish support for DSM function numbers 4, 5, and 6. Recall that the support for _LS{I,R,W} family of methods results in the DIMM being marked as supporting those command numbers at acpi_nfit_register_dimms() time. The DSM function mask is only used for ND_CMD_CALL support of non-NVDIMM_FAMILY_INTEL devices. Fixes: 31eca76 ("nfit, libnvdimm: limited/whitelisted dimm command...") Cc: <stable@vger.kernel.org> Link: pmem/ndctl#78 Reported-by: Sujith Pandel <sujith_pandel@dell.com> Tested-by: Sujith Pandel <sujith_pandel@dell.com> Reviewed-by: Vishal Verma <vishal.l.verma@intel.com> Reviewed-by: Jeff Moyer <jmoyer@redhat.com> Signed-off-by: Dan Williams <dan.j.williams@intel.com>

…inux/kernel/git/nvdimm/nvdimm Pull libnvdimm fixes from Dan Williams: "A fix for namespace label support for non-Intel NVDIMMs that implement the ACPI standard label method. This has apparently never worked and could wait for v5.1. However it has enough visibility with hardware vendors [1] and distro bug trackers [2], and low enough risk that I decided it should go in for -rc4. The other fixups target the new, for v5.0, nvdimm security functionality. The larger init path fixup closes a memory leak and a potential userspace lockup due to missed notifications. [1] pmem/ndctl#78 [2] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1811785 These have all soaked in -next for a week with no reported issues. Summary: - Fix support for NVDIMMs that implement the ACPI standard label methods. - Fix error handling for security overwrite (memory leak / userspace hang condition), and another one-line security cleanup" * tag 'libnvdimm-fixes-5.0-rc4' of git://git.kernel.org/pub/scm/linux/kernel/git/nvdimm/nvdimm: acpi/nfit: Fix command-supported detection acpi/nfit: Block function zero DSMs libnvdimm/security: Require nvdimm_security_setup_events() to succeed nfit_test: fix security state pull for nvdimm security nfit_test

The _DSM function number validation only happens to succeed when the generic Linux command number translation corresponds with a DSM-family-specific function number. This breaks NVDIMM-N implementations that correctly implement _LSR, _LSW, and _LSI, but do not happen to publish support for DSM function numbers 4, 5, and 6. Recall that the support for _LS{I,R,W} family of methods results in the DIMM being marked as supporting those command numbers at acpi_nfit_register_dimms() time. The DSM function mask is only used for ND_CMD_CALL support of non-NVDIMM_FAMILY_INTEL devices. Fixes: 31eca76 ("nfit, libnvdimm: limited/whitelisted dimm command...") Cc: <stable@vger.kernel.org> Link: pmem/ndctl#78 Reported-by: Sujith Pandel <sujith_pandel@dell.com> Tested-by: Sujith Pandel <sujith_pandel@dell.com> Reviewed-by: Vishal Verma <vishal.l.verma@intel.com> Reviewed-by: Jeff Moyer <jmoyer@redhat.com> Signed-off-by: Dan Williams <dan.j.williams@intel.com> Signed-off-by: HouHongxun <houhongxun@gmail.com>

djbw · 2019-01-31T03:39:46Z

Fixed as of kernel v5.0-rc4.

commit 11189c1089da413aa4b5fd6be4c4d47c78968819 upstream. The _DSM function number validation only happens to succeed when the generic Linux command number translation corresponds with a DSM-family-specific function number. This breaks NVDIMM-N implementations that correctly implement _LSR, _LSW, and _LSI, but do not happen to publish support for DSM function numbers 4, 5, and 6. Recall that the support for _LS{I,R,W} family of methods results in the DIMM being marked as supporting those command numbers at acpi_nfit_register_dimms() time. The DSM function mask is only used for ND_CMD_CALL support of non-NVDIMM_FAMILY_INTEL devices. Fixes: 31eca76 ("nfit, libnvdimm: limited/whitelisted dimm command...") Cc: <stable@vger.kernel.org> Link: pmem/ndctl#78 Reported-by: Sujith Pandel <sujith_pandel@dell.com> Tested-by: Sujith Pandel <sujith_pandel@dell.com> Reviewed-by: Vishal Verma <vishal.l.verma@intel.com> Reviewed-by: Jeff Moyer <jmoyer@redhat.com> Signed-off-by: Dan Williams <dan.j.williams@intel.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

commit 11189c1 upstream. The _DSM function number validation only happens to succeed when the generic Linux command number translation corresponds with a DSM-family-specific function number. This breaks NVDIMM-N implementations that correctly implement _LSR, _LSW, and _LSI, but do not happen to publish support for DSM function numbers 4, 5, and 6. Recall that the support for _LS{I,R,W} family of methods results in the DIMM being marked as supporting those command numbers at acpi_nfit_register_dimms() time. The DSM function mask is only used for ND_CMD_CALL support of non-NVDIMM_FAMILY_INTEL devices. Fixes: 31eca76 ("nfit, libnvdimm: limited/whitelisted dimm command...") Cc: <stable@vger.kernel.org> Link: pmem/ndctl#78 Reported-by: Sujith Pandel <sujith_pandel@dell.com> Tested-by: Sujith Pandel <sujith_pandel@dell.com> Reviewed-by: Vishal Verma <vishal.l.verma@intel.com> Reviewed-by: Jeff Moyer <jmoyer@redhat.com> Signed-off-by: Dan Williams <dan.j.williams@intel.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

commit 11189c1089da413aa4b5fd6be4c4d47c78968819 upstream. The _DSM function number validation only happens to succeed when the generic Linux command number translation corresponds with a DSM-family-specific function number. This breaks NVDIMM-N implementations that correctly implement _LSR, _LSW, and _LSI, but do not happen to publish support for DSM function numbers 4, 5, and 6. Recall that the support for _LS{I,R,W} family of methods results in the DIMM being marked as supporting those command numbers at acpi_nfit_register_dimms() time. The DSM function mask is only used for ND_CMD_CALL support of non-NVDIMM_FAMILY_INTEL devices. Fixes: 31eca76ba2fc ("nfit, libnvdimm: limited/whitelisted dimm command...") Cc: <stable@vger.kernel.org> Link: pmem/ndctl#78 Reported-by: Sujith Pandel <sujith_pandel@dell.com> Tested-by: Sujith Pandel <sujith_pandel@dell.com> Reviewed-by: Vishal Verma <vishal.l.verma@intel.com> Reviewed-by: Jeff Moyer <jmoyer@redhat.com> Signed-off-by: Dan Williams <dan.j.williams@intel.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>