Skip to content
This repository has been archived by the owner. It is now read-only.

RBD failed to mount using rbd kernel module: tee: /sys/bus/rbd/add: Invalid argument #2544

Closed
xcompass opened this issue Jan 14, 2019 · 4 comments

Comments

@xcompass
Copy link

@xcompass xcompass commented Jan 14, 2019

Issue Report

Bug

Container Linux Version

NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1995.1.0
VERSION_ID=1995.1.0
BUILD_ID=2019-01-08-0051
PRETTY_NAME="Container Linux by CoreOS 1995.1.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

Environment

What hardware/cloud provider/hypervisor is being used to run Container Linux?
vCloud Director - On Perm

Expected Behavior

Able to mount RBD volume successfully

Actual Behavior

When running

echo 'CEPH_MON_IP name=admin,secret=SECRET_KEY  rbd myrbdvolume' | sudo tee /sys/bus/rbd/add

I got error:

tee: /sys/bus/rbd/add: Invalid argument

Reproduction Steps

Need to have a Ceph cluster.

  1. Login to CoreOS beta 1995.1.0
  2. Run command
modprobe rbd
echo 'CEPH_MON_IP name=admin,secret=SECRET_KEY  rbd myrbdvolume' | sudo tee /sys/bus/rbd/add

This should yield the error.

Other Information

The command works in other versions. The versions I tested are

  • Container Linux by CoreOS stable (1745.3.1)
  • Container Linux by CoreOS stable (1911.4.0)
  • Container Linux by CoreOS stable (1967.3.0)

Here are the modinfo rbd for 1967.3.0

filename:       /lib/modules/4.14.88-coreos/kernel/drivers/block/rbd.ko
license:        GPL
description:    RADOS Block Device (RBD) driver
author:         Jeff Garzik <jeff@garzik.org>
author:         Yehuda Sadeh <yehuda@hq.newdream.net>
author:         Sage Weil <sage@newdream.net>
author:         Alex Elder <elder@inktank.com>
depends:        libceph
intree:         Y
name:           rbd
vermagic:       4.14.88-coreos SMP mod_unload
sig_id:         PKCS#7
signer:
sig_key:
sig_hashalgo:   md4
signature:      30:82:02:B2:06:09:2A:86:48:86:F7:0D:01:07:02:A0:82:02:A3:30:
                82:02:9F:02:01:01:31:0D:30:0B:06:09:60:86:48:01:65:03:04:02:
                01:30:0B:06:09:2A:86:48:86:F7:0D:01:07:01:31:82:02:7C:30:82:
                02:78:02:01:01:30:53:30:46:31:14:30:12:06:03:55:04:0A:0C:0B:
                43:6F:72:65:4F:53:2C:20:49:6E:63:31:2E:30:2C:06:03:55:04:03:
                0C:25:4D:6F:64:75:6C:65:20:73:69:67:6E:69:6E:67:20:6B:65:79:
                20:66:6F:72:20:34:2E:31:34:2E:38:38:2D:63:6F:72:65:6F:73:02:
                09:00:F3:29:EE:EC:DF:58:5D:B4:30:0B:06:09:60:86:48:01:65:03:
                04:02:01:30:0D:06:09:2A:86:48:86:F7:0D:01:01:01:05:00:04:82:
                02:00:6D:C8:25:14:E4:FE:B3:D9:FF:ED:2B:9B:31:00:B7:D9:E5:57:
                02:7F:E7:2F:94:D6:60:2B:97:AE:35:AB:FA:64:8C:B6:DF:E5:E9:25:
                B4:10:06:D5:48:4D:8D:2D:68:27:49:E7:B8:97:56:70:55:A4:E9:04:
                0A:C0:39:8B:97:34:0F:DD:3B:E9:D5:8F:43:A3:FB:13:3D:45:6B:47:
                EC:F1:5E:80:70:67:CC:A1:84:D4:DB:E2:44:27:A6:2D:DA:38:BE:94:
                F7:70:91:80:64:74:41:1D:E3:30:8B:DA:78:80:03:6D:0B:56:1D:8A:
                80:39:70:8E:F7:B1:A9:3B:DA:13:09:88:D6:03:87:ED:B5:96:64:0F:
                27:97:1A:F6:94:70:22:1E:9B:53:D7:D4:12:B1:C3:AE:ED:CD:5E:62:
                FD:03:12:B0:68:30:1B:3F:32:58:70:3C:4A:2A:BF:AB:28:A2:36:CC:
                69:AE:57:BE:9A:81:B7:99:7B:04:B8:BE:5B:54:62:B2:1E:88:CA:A8:
                6E:22:0B:B6:EE:60:38:F9:BE:D8:05:3C:15:D1:70:B8:CA:EA:88:C4:
                92:FB:C0:1B:1E:4C:8F:89:55:3C:47:80:F7:1C:64:5B:49:61:9D:96:
                A9:28:78:23:DA:98:5B:B0:53:3D:90:44:E9:14:2E:70:97:7B:8D:5A:
                32:83:49:D2:E7:99:78:9E:F4:C4:65:32:D2:E1:0C:3A:E8:06:A6:EF:
                33:A1:1E:E0:14:C0:FA:A8:88:8B:E3:9C:1E:6A:FF:93:9D:7C:97:C7:
                19:FE:33:47:E5:2E:C6:82:9E:C2:77:1F:24:E3:09:E1:EE:2D:BA:25:
                77:A3:F8:83:35:36:A8:06:64:BF:07:98:2C:F8:92:B3:FF:82:37:DA:
                21:F2:C7:4F:F9:82:28:BE:6B:DA:14:2C:44:9D:23:F1:D6:A8:40:6F:
                F6:56:F7:1F:11:15:DE:FC:C7:F4:69:ED:94:0D:F0:EE:33:B7:AF:65:
                E2:17:E5:6D:0B:CE:37:83:5A:AC:A5:6F:D1:A4:CC:DD:53:44:DA:13:
                01:07:33:C2:36:C1:83:5B:66:CA:E1:5F:22:B6:92:79:C0:D2:CC:14:
                D1:95:8C:97:E5:18:3B:D2:A8:F8:19:A9:2E:BC:34:E6:95:05:EE:5F:
                46:5C:23:51:6C:D4:7C:1E:73:26:D4:B4:82:2E:84:77:34:23:B0:CB:
                53:8A:6A:3E:9D:0B:8E:FE:27:63:CE:F5:80:F3:84:71:85:EF:AE:C5:
                47:EC:45:C8:AF:D3:4C:A4:1C:46:15:2C:BB:A5:02:36:F3:ED:A2:AB:
                E3:70:A1:62:EC:1A:E5:45:EC:BB:F9:9F:96:47
parm:           single_major:Use a single major number for all rbd devices (default: false) (bool)

This is for 1995.1.0

filename:       /lib/modules/4.19.13-coreos/kernel/drivers/block/rbd.ko
license:        GPL
description:    RADOS Block Device (RBD) driver
author:         Jeff Garzik <jeff@garzik.org>
author:         Yehuda Sadeh <yehuda@hq.newdream.net>
author:         Sage Weil <sage@newdream.net>
author:         Alex Elder <elder@inktank.com>
depends:        libceph
intree:         Y
name:           rbd
vermagic:       4.19.13-coreos SMP mod_unload
sig_id:         PKCS#7
signer:
sig_key:
sig_hashalgo:   md4
signature:      30:82:02:B2:06:09:2A:86:48:86:F7:0D:01:07:02:A0:82:02:A3:30:
                82:02:9F:02:01:01:31:0D:30:0B:06:09:60:86:48:01:65:03:04:02:
                01:30:0B:06:09:2A:86:48:86:F7:0D:01:07:01:31:82:02:7C:30:82:
                02:78:02:01:01:30:53:30:46:31:14:30:12:06:03:55:04:0A:0C:0B:
                43:6F:72:65:4F:53:2C:20:49:6E:63:31:2E:30:2C:06:03:55:04:03:
                0C:25:4D:6F:64:75:6C:65:20:73:69:67:6E:69:6E:67:20:6B:65:79:
                20:66:6F:72:20:34:2E:31:39:2E:31:33:2D:63:6F:72:65:6F:73:02:
                09:00:D2:7F:34:D1:90:A2:06:7E:30:0B:06:09:60:86:48:01:65:03:
                04:02:01:30:0D:06:09:2A:86:48:86:F7:0D:01:01:01:05:00:04:82:
                02:00:8A:8E:BB:9B:2C:1A:54:32:F8:B4:61:C6:46:16:3B:F7:D3:4C:
                98:75:49:27:A9:41:DF:9C:39:F1:06:0A:3F:05:2E:01:B0:54:E1:45:
                06:70:23:36:90:BC:5F:28:3F:9E:BC:AA:AA:40:17:B8:71:18:70:EA:
                A0:22:2A:D6:92:B7:9F:CA:AA:6D:DC:14:5C:4E:57:F2:F3:9E:07:47:
                09:6A:E5:B7:92:8B:DE:F2:03:0F:E9:7D:E3:0E:21:6C:F5:83:C4:3F:
                D1:11:9A:97:23:F9:57:9D:5B:C3:50:83:CC:6F:22:19:1D:60:98:71:
                BC:0C:E6:FE:76:A0:42:CD:84:02:0C:22:C3:6C:5E:5D:03:8F:B8:05:
                55:D1:8E:E6:4C:79:1B:1F:DC:81:E3:35:B9:62:8F:75:9B:BE:90:DD:
                98:2B:95:EE:8C:7F:7D:C4:FD:74:7A:15:A0:03:21:A5:ED:F9:5E:65:
                89:AF:54:6E:D1:D9:6E:44:EA:03:AD:4D:C8:C6:36:A8:1D:B4:EC:7D:
                8A:8D:AF:F0:AF:EF:13:F0:80:F8:27:A6:E2:99:FA:D9:8D:CD:92:56:
                B2:6B:07:9D:E9:81:F9:99:40:A8:BF:9B:F8:A9:46:3A:F2:90:BC:4F:
                52:8F:52:AE:11:4A:DB:93:E4:C9:BD:2F:DD:CB:24:01:13:67:ED:37:
                3A:5F:D3:3C:B0:14:A4:3D:92:8B:F7:12:81:42:04:4E:10:73:3F:6F:
                42:24:5F:79:E8:F3:7A:E1:80:A3:33:17:F6:E4:2C:F7:9C:3D:2F:CD:
                3C:1B:F6:98:91:F0:19:82:9D:A6:0C:45:9D:85:44:79:E8:A6:5C:70:
                5D:95:B5:00:E6:04:CD:31:DE:0D:ED:50:E8:83:37:F1:F1:66:2E:A0:
                8D:68:F2:C9:ED:21:67:BF:EA:45:24:10:3D:44:F5:CA:6F:D0:4F:0B:
                40:67:2C:CD:CA:04:C8:6B:70:B8:D8:71:04:C5:8A:9E:82:75:95:1A:
                40:7B:83:B7:26:A9:60:D6:F4:23:1B:D5:41:9E:12:46:F3:52:77:5F:
                3E:B9:DF:67:6D:4C:07:A4:3E:39:C5:E6:A3:1C:8B:78:8E:A9:69:96:
                40:0F:F1:39:B0:75:F7:35:17:33:08:2D:1E:68:12:2B:AA:69:90:C6:
                23:91:5B:0D:F8:14:6D:45:EF:5F:BE:E4:47:52:B1:2B:38:A0:5A:18:
                86:14:12:54:53:A8:14:00:0D:40:FA:3B:B2:CC:9B:65:03:FA:61:2F:
                E5:A8:BE:90:16:B7:B2:4A:50:A2:03:AE:2B:F0:D3:3C:1E:9C:28:DF:
                62:AC:43:75:BB:42:FA:CD:A8:AB:A2:40:9E:9D
parm:           single_major:Use a single major number for all rbd devices (default: true) (bool)
@bgilbert
Copy link
Member

@bgilbert bgilbert commented Jan 22, 2019

This seems likely to be a regression in the 4.19 kernel.

@dm0-
Copy link

@dm0- dm0- commented Feb 1, 2019

Linux 4.15 defaulted single_major to true in torvalds/linux@3cfa3b1, which means you should use /sys/bus/rbd/add_single_major now. You can revert to the old behavior with e.g. modprobe rbd single_major=0.

@dm0- dm0- closed this Feb 1, 2019
@dm0-
Copy link

@dm0- dm0- commented Feb 1, 2019

Since Container Linux does not include the userspace utilities that normally handle this behavior change, we'll revert 4.19 to the previous default behavior before it reaches the stable channel.

@bgilbert
Copy link
Member

@bgilbert bgilbert commented Feb 12, 2019

This is fixed in alpha 2051.0.0 and beta 2023.2.0. Thanks for reporting.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants