daxctl always map system-ram into target_node 0

[mesunyyam]

Hello,

I am trying to change devdax dax0.0 device to PMEM only NUMA node by commanding as below.

$ daxctl reconfigure-device -m system-ram --no-online dax0.0

However, this system-ram is always mapped into target node 0 and created system-ram is always immediately online even though I specifically says "--no-online" in option.
Desirable results are that newly created system-ram is mapped to target node 2 (since my server has 2 node) and this should be offline.

I checked kernel configuration as follows.

CONFIG_DEV_DAX_KMEM=m, CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE is not set, NUMA_BALANCING=y

Is there another step that I need to do to solve this issue? (e.g. BIOS setting or additional kernel configurations?)

Thanks,
Miseon Han

[mesunyyam]

Hello,

I solved the one issue that system-ram is immediately mapped into online memory.
I modified udev rules and reload rules by following commands.

$ vim /etc/udev/rules.d/filename.rules
$ udevadm control --reload-rules
$ udevadm trigger --type=devices --action=change

However, I sill have the remain issue that system-ram is always mapped into target_node 0 (can't make PMEM only node).
I am planning to look into BIOS setup.

[NoufalKMuhammed]

Hi @mesunyyam,
In a VMware virtual machine setup, I am also having the issue "system-ram is always mapped into target_node 0 (can't make PMEM only node)". Did you get a solution for this?.

[Yitrus]

I'm using Ubuntu 18.04.6 LTS with kernel version 5.3.1-multiclock, which comes from SyLab and just to make sure that the problem isn't caused by the kernel version, I'm getting the same error with version 5.15.17 of the kernel.

$ ndctl -v
66+

A while back, I used this kernel to successfully configure persistent memory as a separate NUMA node on a DELL R750 server following the steps. Now I am going to emulate persistent memory on a PC (with a 32G memory bank) and configure the emulated persistent memory as a separate NUMA node as well. Following the steps I set:
GRUB_CMDLINE_LINUX="memmap=24G!8G"

I got the following message after booting:

$ egrep '(DAX|PMEM)' /boot/config-`uname -r`
CONFIG_X86_PMEM_LEGACY_DEVICE=y
CONFIG_X86_PMEM_LEGACY=y
CONFIG_VIRTIO_PMEM=m
CONFIG_BLK_DEV_PMEM=m
CONFIG_NVDIMM_DAX=y
CONFIG_DAX_DRIVER=y
CONFIG_DAX=y
CONFIG_DEV_DAX=m
CONFIG_DEV_DAX_PMEM=m
CONFIG_DEV_DAX_KMEM=m
CONFIG_DEV_DAX_PMEM_COMPAT=m
CONFIG_FS_DAX=y
CONFIG_FS_DAX_PMD=y
CONFIG_ARCH_HAS_PMEM_API=y

$ ndctl list -N
[
  {
    "dev":"namespace0.0",
    "mode":"fsdax",
    "map":"mem",
    "size":25769803776,
    "sector_size":512,
    "blockdev":"pmem0"
  }
]

$ numactl -H
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
node 0 size: 7698 MB
node 0 free: 1933 MB
node distances:
node   0 
  0:  10 

Then I did the following:

$ sudo ndctl destroy-namespace namespace0.0 --force
destroyed 0 namespaces

$ sudo ndctl create-namespace --mode=devdax --map=mem
{
  "dev":"namespace0.0",
  "mode":"devdax",
  "map":"mem",
  "size":"24.00 GiB (25.77 GB)",
  "uuid":"5d7ff602-47c3-426c-9e26-80a5bc183ee7",
  "daxregion":{
    "id":0,
    "size":"24.00 GiB (25.77 GB)",
    "align":2097152,
    "devices":[
      {
        "chardev":"dax0.0",
        "size":"24.00 GiB (25.77 GB)",
        "target_node":0,
        "mode":"devdax"
      }
    ]
  },
  "align":2097152
}

$ daxctl list
[
  {
    "chardev":"dax0.0",
    "size":25767706624,
    "target_node":0,
    "mode":"devdax"
  }
]

daxctl always map system-ram into target_node 0

If I try to do the next step：

$ daxctl reconfigure-device dax0.0 --mode=system-ram
libdaxctl: daxctl_dev_disable: dax0.0: error: device model is dax-class
dax0.0: disable failed: Operation not supported
error reconfiguring devices: Operation not supported
reconfigured 0 devices

Since node 0 exists and is being used by the system, I'm guessing that the target_node conflict is the cause of this error.

After that I set up 3 namespaces in kernel 5.15.17 by commanding :GRUB_CMDLINE_LINUX="memmap=4G!4G memmap=4G!16G memmap=4G!28G"

$ ndctl list -N
[
  {
    "dev":"namespace1.0",
    "mode":"fsdax",
    "map":"mem",
    "size":4294967296,
    "sector_size":512,
    "blockdev":"pmem1"
  },
  {
    "dev":"namespace0.0",
    "mode":"fsdax",
    "map":"mem",
    "size":4294967296,
    "sector_size":512,
    "blockdev":"pmem0"
  },
  {
    "dev":"namespace2.0",
    "mode":"fsdax",
    "map":"mem",
    "size":4294967296,
    "sector_size":512,
    "blockdev":"pmem2"
  }
]

I deleted each of these three and re-created them in devdax mode, and they are all still target_node 0.

$ daxctl list
[
  {
    "chardev":"dax1.0",
    "size":4292870144,
    "target_node":0,
    "mode":"devdax"
  },
  {
    "chardev":"dax2.0",
    "size":4292870144,
    "target_node":0,
    "mode":"devdax"
  },
  {
    "chardev":"dax0.0",
    "size":4292870144,
    "target_node":0,
    "mode":"devdax"
  }
]