Starting iscsi service yields "ConditionDirectoryNotEmpty=/var/lib/iscsi/nodes was not met" error #219
Comments
|
I'm not sure I can help. Perhaps @cleech can provide insight? The open-iscsi setup on SUSE doesn't have a condition on the "nodes" directory being empty. And I'm not sure why SELinux would change the condition checking, but I suspect that's a systemd bug (or quirk). |
|
Is the nodes directory empty? The iscsi service is really about auto-starting iscsi sessions from those records. The iscsid should start on demand as the iscsid.socket unit should be listening, so you can go ahead and issue iscsiadm commands without iscsi.service running. |
|
The nodes directory is empty. As yet, I haven't been able to establish any iscsi sessions. [root@localhost nodes]# iscsiadm --mode discovery -t sendtargets --portal 192.168.1.10:3260 The coredump as it appears in journalctl is : |
|
OK, that i haven't seen before. It looks like there's an overflow checking the CHAP mode configurations. Have you configured CHAP settings? Could you share the node.session.auth.chap_algs part of /etc/iscsid.conf? |
|
The CHAP algorithms are all the default settings - ie. commented out when the RPM is installed. *************CHAP Settings*************To enable CHAP authentication set node.session.auth.authmethodto CHAP. The default is None.#node.session.auth.authmethod = CHAP To configure which CHAP algorithms to enable setnode.session.auth.chap_algs to a comma seperated list.The algorithms should be listen with most prefered first.Valid values are MD5, SHA1, SHA256, and SHA3-256.The default is MD5.#node.session.auth.chap_algs = SHA3-256,SHA256,SHA1,MD5 To set a CHAP username and password for initiatorauthentication by the target(s), uncomment the following lines:#node.session.auth.username = username To set a CHAP username and password for target(s)authentication by the initiator, uncomment the following lines:#node.session.auth.username_in = username_in To enable CHAP authentication for a discovery session to the targetset discovery.sendtargets.auth.authmethod to CHAP. The default is None.#discovery.sendtargets.auth.authmethod = CHAP To set a discovery session CHAP username and password for the initiatorauthentication by the target(s), uncomment the following lines:#discovery.sendtargets.auth.username = username To set a discovery session CHAP username and password for target(s)authentication by the initiator, uncomment the following lines:#discovery.sendtargets.auth.username_in = username_in |
|
Are you still having this issue? The "ConditionDirectoryNotEmpty" is normal. That's a new thing added by RH that creates the initiator name if it isn't present. So seeing that message just means you already have an initiator name. The iscsiadm core dump is the problem though. Can you run it with "-d8" and capture the output? Before you do that, add "-d8" to the iscsid command line in iscsid.service and reload the service file and restart iscsid. So you should be getting massive debugging from iscsid as well as from iscsiadm. Note: the interface here for comments uses markdown, so be sure to quote any log file data you insert so it doesn't turn out strange (like your last comment did). |
|
yes I'm still having the issue. I've also noticed that it is now started happening on my second fedora box. Obviously some change to iscsid that was introduced. I've run iscsiadm with the -d8 parameter in the service iscsid (after reloading and restarting the service). Here is the journal output :
|
|
The above debugging shows several things:
It looks like you are running:
You need to add "-d8" to this line. And you need to remove the "-d8" from the systemd service file where you added it before (for iscsid). It looks like iscsiadm is core dumping in idbm_recinfo_discovery(). I'm suspecting something is strange about your targetj(s). Can you share info about them? Are they hardware or software targets? What is the CHAP config of the targets? Any chance you could get a ethernet trace? |
|
The hardware target is to a Synology DS1513+ . I have no problems with iscsi when I use it with Virtualbox. Here is the output (it's long). It appears to be discovering all the targets before finally crashing. ``[fedora@localhost system]$ sudo iscsiadm -m discovery -t sendtargets -p 192.168.1.10 -d8 > ~/t.txt |
|
I just had a second look at the output of the log. Iscsiadm seems to have aborted processing targetname "Target-911data" (target 23 of 31) during discovery. Have tried changing the network binding on the target (the Synology NAS), however this made no difference. I don't have any CHAP authentication on any of the Targets. |
|
I'm thinking you are running up against some system limit, though I've never seen this before. How much RAM do you have on your system? Can you run "ulimit -a" and include the output? I'm also guessing this will "go away" (not show) if you have less targets. I have tested before with 100 targets (each fairly small, since I'm using software targets), but not recently. This is exacerbated by the fact that each target has 4 addresses, i.e. 2 ports, and each port has IPv6 and IPv4. You could also try disabling IPv6 (or IPv4) on your system, to cut the number of portals in half. All of this would help me understand what's happening. Also, if this is on RedHat, you should try to engage somebody there (like @cleech, co-maintainer with me). Unless you are able to compile open-iscsi yourself from sources, I can't help you much, since I can't add debugging code and have you run it. |
|
Synology uses Busybox as its Operation System. Here is the output of "ulimit -a". root@DiskStation:~# ulimit -a core file size (blocks, -c) unlimited I just checked the hardware specs for my DS1513+ , and yes there are hard limits for iscsi. targets : 32luns : 256I do have root access to the Synology. Can these limits be changed, or are they set in stone? |
|
ok. I have just done some house cleaning on my Synology, as there were a number of targets not pointing to existing luns. It's now able to discover all the targets without crashing. My previous question still remains (re: iscsi limits - can they be modified?) . Also, can the iscsi daemon server be modified so that it only communicates via ipv4? (I have disabled ipv6 on the Synology nics for the time being.) |
|
Thanks for providing all that debug info. I have something reproducing now, hopefully I'll have a fix pretty quickly. |
|
I think the problem is with the code to setup configuration for the CHAP algorithm parameter. The new code I added for managing a list of negotiation options didn't clear the string first, and during discovery the same recinfo struct is used repeatedly. Every time it's re-initialized the parameter string wasn't cleared and was appended to, until with enough targets discovered at once there's a strcat buffer overflow. |
|
Would you mind trying this test RPM? http://people.redhat.com/cleech/iscsi-f32-discovery/ It just adds this single line change, but I should probably see if I can add some safety limits to the strcat call. |
|
I just tried adding a new target, then using the distro version of iscsiadm to discover the targets, and I got a 'buffer overflow' error. However, after applying the RPM from your test repository, iscsiadm was able to discover all the targets without error. |
I have been trying to get to the bottom of this issue. However so far I haven't had a satisfactory resolution.
I am running Fedora 32 Server and have the latest available RPM version installed. I have checked the iscsi config against a working install, and the config is exactly the same. Root has 755 access to /var/lib/iscsi/nodes directory, with SeLinux managing the directory with iscsi_var_lib_t . The only difference I can think of (whether this makes a difference or not) is that the server has two Nics (1G ethernet and WLan).
Any advice, much appreciated.
The text was updated successfully, but these errors were encountered: