segfault at 0 ip (null) sp 00007fffd345bbe8 error 14 in libgcc_s-4.8.5-20150702.so.1[7faa11c3c000+15000] #654
Comments
|
I have tried the above config on my system and it doesn't segfault. It appears that there is a problem specifically with CentOS at the moment (see issue #650). Since I am unable to reproduce the problem, I need further information in order to be able to investigate the issue. Could you please see what I wrote in issue #650 regarding producing a stacktrace, and post a stacktrace output in this issue. With that information it should give us a much better idea where the problem lies. |
|
Thank you for your reply。 I didn't have this problem when I was installing another machine. This time before I installed Keepalived, I was fully prepared, first of all, to do some of the following installations: |
|
It shouldn't be necessary to install any of those packages, although they might cause some dependent packages to be installed that keepalived does require. Could you provide a list of any dependent packages that were installed when you installed the above 3 packages. I think it would be useful to look at the rpm spec file to see whether it is missing any requires. |
|
I had a look at my CentOS 7.4 VM and openssl-libs was installed as part of the basic install, and I don't have popt or ipvsadm installed on my system so they shouldn't be needed. I've now run your configuration in my CentOS 7.4 VM with the CentOS keepalived v1.3.5 package, and I don't experience a segfault. |
|
Closing since the problem isn't happening any more. If the problem starts happening again, please update this issue. |
|
I'm running CentOS 7.4.1708 (Core) I get the following error when starting Kleepalived. The issue started when we did an update to CentOS and Keepalived: Oct 17 12:22:55 li1029-169 Keepalived[16499]: Starting Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2 here's our config file: vrrp_script chk_haproxy { vrrp_instance VI_210 { Was there ever a fix for this issue? |
|
It seems that for keepalived v1.3.5 to work on CentOS selinux-policy-3.13.1-166.el7_4.4 or later is required (see issue #650). Can you check what version of selinux-policy you have installed. |
|
Thanks, pqarmitage. We don't have SELinux installed. I should have noted that in my post... |
|
Are you running keepalived in a container, since that can cause problems? Otherwise what will be needed is a coredump following the segfault. Running keepalived with the |
|
I'm not running keepalived in a container. |
|
pqarmitage, I followed your notes from issue #650 to create the backtrace: [root@li1029-169 /]# gdb keepalived core.3745 thanks, -K |
|
@kkraemergit that's very helpful. That is precisely the same stack trace as at #650 (comment). The cause and resolution are described in #650 (comment), but essentially mean either taking a slightly later version of the source code, or ensuring that the iptables and ipsets kernel modules are loaded before running keepalived. The simplest wa y to ensure that the modules are loaded would be to run the following: |
|
Hi, pqarmitage. I thought my issue was the same as 650. But, I ran the commands -nvL INPUT; ip6tables -nvL INPUT; ipset list; and see the output of iptables entries expected. Then I started keepalived and got a Segmentation fault. Here's the backtrace after running the iptables commands above. [root@li1029-169 /]# gdb keepalived core.30697 Shall I try to install keepalived commit f7cd991 per issue 650? Best regards, -K |
|
Yes adding commit f7cd991 would be worth trying, or better still commit 47b5e44 since that includes additional commits related to failing to load modules. If neither of them works then using the latest commit in master would be worth a try. Do you know if there is any way using CentOS that we can get it to report source code lines numbers in the backtrace? Fedora does print line numbers which makes it much easier to identify exactly where it is segfaulting. |
|
I'll try commit 47b5e44 I tried to get line numbers for you, but not much luck. Could "remove_ipsets+22" be line 22 below? (gdb) info line *0x0000556a6f434216 BR -K |
|
The +22 is the byte offset of the instruction pointer from the start of remove_ipsets. If you build your own version of keepalived, you should be able to run an unstripped executable, in which was we might get some more useful information. |
|
It seems as though |
|
Thanks for finding "debuginfo-install keepalived-1.3.5-1.el7.x86_64" Here's the updated trace [root@li1029-169 /]# gdb keepalived core.30697 fini=, rtld_fini=, stack_end=0x7ffcbb303f18) at ../csu/libc-start.c:274 #8 0x0000556a6f404e6e in _start () (gdb) |
|
I really don't know it this has got anything to do with the issue, but what version of libipset do you have installed? |
|
Does this help determine version of libipset installed? [root@li1029-169 /]# yum info ipset
|
|
Ok, so that's not the issue. What I can't understand is what is the difference between what you are running and what I am doing. I see that you have a linode repo enabled; are you by any chance running keepalived in a linode VM? There have been some problems with that in the past, although I think building from commit 47b5e44 resolved it. |
|
It's a linode VM. Keepalived was working well up until our last update yesterday. I can give you a dump of what was updated. Would that help? |
|
I'll install commit 47b5e44. Thanks for all your help. I hope some of this was useful. Attached is what was updated. Keepalived worked prior to this update. BR, -K |
|
I see keepalived was updated, and if I remember correctly the previous version was v1.2.13. That was over three years of updates in more than 1100 commits. I think the problems have all related to not using the new selinux-policy, running keepalived in containers where it can't load modules into the kernel which has to occur outside the container, and using the Linode kernel in a VM, when keepalived didn't cope with the iptables and ipsets not being in loadable modules. |
|
pqarmitage, we've installed commit 47b5e44 on Linode VM, centos 7, and successfully tested fail-over. Thanks for your help. BR -K |
|
Registering Kernel netlink reflector can help me? thank you! |
|
@lishuai2016 - Could you please raise this as a new issue, and include the following information: output of Problems like this are usually caused by running keepalived in a container and the host doesn't have the necessary modules loaded such as ip_vs, ip_set, ip_table, ip6_table and the container doesn't have sufficient privileges to load the modules. |
Sep 20 10:06:21 db2 systemd: Starting LVS and VRRP High Availability Monitor...
Sep 20 10:06:21 db2 Keepalived[2961]: Starting Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
Sep 20 10:06:21 db2 Keepalived[2961]: Unable to resolve default script username 'keepalived_script' - ignoring
Sep 20 10:06:21 db2 Keepalived[2961]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 20 10:06:21 db2 Keepalived[2962]: Starting Healthcheck child process, pid=2963
Sep 20 10:06:21 db2 Keepalived[2962]: Starting VRRP child process, pid=2964
Sep 20 10:06:21 db2 Keepalived_vrrp[2964]: Registering Kernel netlink reflector
Sep 20 10:06:21 db2 Keepalived_vrrp[2964]: Registering Kernel netlink command channel
Sep 20 10:06:21 db2 Keepalived_vrrp[2964]: Registering gratuitous ARP shared channel
Sep 20 10:06:21 db2 Keepalived_vrrp[2964]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 20 10:06:21 db2 Keepalived_vrrp[2964]: Unknown keyword 'smtp_connection_timeout'
Sep 20 10:06:21 db2 systemd: Started LVS and VRRP High Availability Monitor.
Sep 20 10:06:21 db2 kernel: nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
Sep 20 10:06:21 db2 kernel: IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
Sep 20 10:06:21 db2 kernel: IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
Sep 20 10:06:21 db2 kernel: IPVS: Creating netns size=2040 id=0
Sep 20 10:06:21 db2 kernel: IPVS: ipvs loaded.
Sep 20 10:06:21 db2 Keepalived_healthcheckers[2963]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 20 10:06:21 db2 Keepalived_healthcheckers[2963]: Unknown keyword 'smtp_connection_timeout'
Sep 20 10:06:21 db2 Keepalived_healthcheckers[2963]: Unknown keyword 'nat_mask'
Sep 20 10:06:21 db2 Keepalived_healthcheckers[2963]: persistent_timeout invalid
Sep 20 10:06:21 db2 Keepalived_healthcheckers[2963]: Unknown keyword 'nb_get_retry'
Sep 20 10:06:21 db2 Keepalived_healthcheckers[2963]: Activating healthchecker for service [10.10.10.200]:8082
Sep 20 10:06:21 db2 kernel: IPVS: [wrr] scheduler registered.
Sep 20 10:06:26 db2 Keepalived_vrrp[2964]: VRRP_Instance(VI_1) removing protocol VIPs.
Sep 20 10:06:26 db2 kernel: keepalived[2964]: segfault at 0 ip (null) sp 00007fffd345bbe8 error 14 in libgcc_s-4.8.5-20150702.so.1[7faa11c3c000+15000]
Sep 20 10:06:26 db2 Keepalived[2962]: Keepalived_vrrp exited due to segmentation fault (SIGSEGV).
Sep 20 10:06:26 db2 Keepalived[2962]: Please report a bug at https://github.com/acassen/keepalived/issues
Sep 20 10:06:26 db2 Keepalived[2962]: and include this log from when keepalived started, what happened
Sep 20 10:06:26 db2 Keepalived[2962]: immediately before the crash, and your configuration file.
Sep 20 10:06:26 db2 Keepalived[2962]: VRRP child process(2964) died: Respawning
Sep 20 10:06:26 db2 Keepalived[2962]: Starting VRRP child process, pid=2983
Sep 20 10:06:26 db2 Keepalived_vrrp[2983]: Registering Kernel netlink reflector
Sep 20 10:06:26 db2 Keepalived_vrrp[2983]: Registering Kernel netlink command channel
Sep 20 10:06:26 db2 Keepalived_vrrp[2983]: Registering gratuitous ARP shared channel
Sep 20 10:06:26 db2 Keepalived_vrrp[2983]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 20 10:06:26 db2 Keepalived_vrrp[2983]: Unknown keyword 'smtp_connection_timeout'
Sep 20 10:06:26 db2 Keepalived_vrrp[2983]: VRRP_Instance(VI_1) removing protocol VIPs.
Sep 20 10:06:26 db2 kernel: keepalived[2983]: segfault at 0 ip (null) sp 00007fffd345bb68 error 14 in libgcc_s-4.8.5-20150702.so.1[7faa11c3c000+15000]
......
system: centOS 7.1
==keepalived.conf==
! Configuration File for keepalived
global_defs {
notification_email {
syic@tom.com
}
notification_email_from syic@tom.com
smtp_server 10.10.10.1
smtp_connection_timeout 30
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state MASTER
interface ens192
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.10.10.200
}
}
virtual_server 10.10.10.200 8082 {
delay_loop 6
lb_algo wrr
lb_kind DR
nat_mask 255.255.255.0
persistence_timeout 0
protocol TCP
real_server 10.10.10.33 8082 {
weight 3
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 8082
}
}
}
The text was updated successfully, but these errors were encountered: