Problems With LVM Mounts in CentOS 7 with Snoopy Enabled

[jwbrown77]

Hi Bostjan,

As we had discussed in previous items, I am attempting to get Snoopy working properly in CentOS 7. I thought I had figured it out. After successfully building from the RPM spec and installing, I was able to confirm that commands were being properly.

However, upon restart, the system fails to boot because it claims it is timing out waiting for one of the LVM block devices; in my case, the device for my /var partition.

I have debugged my setup extensively and can confirm this is recreatable only when I enable snoopy. After this crash, I can reboot back into rescue mode, disable snoopy, then the system will boot again.

Please see this screen capture of my VM console: http://i.imgur.com/DHmhsI3.png

I'm unsure on how to proceed from here. Do you have any tips on what I can do to remedy? Let me know if you'd like my kickstart file, though I'm guessing you'll be able to recreate on any CentOS 7 install.

Thanks

[jwbrown77]

Hi Bostjan,

For what it's worth, I figured I should try a very basic install without any of my custom configurations to ensure it's not my modifications that are causing issues.

I installed CentOS 7 with all the default settings. After install, I only installed the tools required to build the SPEC against snoopy 2.1.0. I then installed it using "yum localinstall". On reboot, it does indeed do the same thing as I posted above.

I just wanted to be sure I wasn't wasting your time when the problem was on my end; but it does appear to be some sort of fundamental conflict with snoopy and CentOS 7.

Thanks

[bostjan]

It very much reminds me about this issue, which was resolved by finding and closing unintentionally open log descriptor.

Are you using config file? Any configure flags?

b.

[bostjan]

Yeah, I see it!
Thought ATM I haven't got the foggiest about what is going on.

b.

[bostjan]

It seems this bug is not related to LVM - I tried it the second time on a CentOS 7 which I manually installed on ordinary partitions - no difference.

The first impression is that it is systemd related, but I think that is not the case.
I believe it must be certain command that is failing and thus causing the loop/delay/hang.

Unfortunately I can not give an ETA about when I will have time to look into this.

PS: This is the first bite from systemd - debugging is way harder than with sequential init systems.

[jwbrown77]

Hi Bostjan,

Thanks. I'm always happy with simply having another person confirm what I'm seeing. 😄

For what it's worth, I'm using the default configuration file, and the same configure flags as before (automatically generated from the RPM spec):

./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info

Thanks

[bostjan]

Finally I've figured this one out (I think).

This is snoopy-journald issue.

For actual logging, snoopy uses syslog() syscall (which sends message to /dev/log socket, where system logger listens for it). On non-systemd OSes, if something happens (system logger not running), message is simply discarded (or it gets sent to console).

Systemd+journald behave quite differently. It pre-creates /dev/log socket, and anything sent to it is buffered until logging daemon (in this case journald) becomes available and consumes buffered messages. Unfortunately this buffer is quite small (10 entries). Once the buffer fills up, systemd's pre-created /dev/log socket starts blocking and all processes, which use normal syslog() logging method, get hanged on it.

It seems that even starting journald causes some messages (probably generated by snoopy) to be sent to /dev/log, which now blocks journald's startup. There it is, a nice deadlock.

I am still investigating this issue, but it seems systemd needs some special babysitting.

Some references:
https://lists.balabit.hu/pipermail/syslog-ng/2013-September/020726.html

[bostjan]

Fixed in 2.2.1, just tested it, works :)

b.

[jwbrown77]

Thanks Bostjan!

[neubian]

I never knew why I had to reinstall a server from scratch earlier this year, because LVM seemed perfectly healthy. I'd installed snoopy just before. Good to know.