[dev.icinga.com #615] Interfacechecks switches to UNKNOWN state after restart of ICINGA #310

Closed
icinga-migration opened this Issue Jul 10, 2010 · 12 comments

Comments

Projects
None yet
1 participant
Member

icinga-migration commented Jul 10, 2010

This issue has been migrated from Redmine: https://dev.icinga.com/issues/615

Created by crsp on 2010-07-10 14:48:39 +00:00

Assignee: mfriedrich
Status: Closed (closed on 2011-01-10 17:20:24 +00:00)
Target Version: (none)
Last Update: 2011-01-10 17:20:24 +00:00 (in Redmine)


As requested by Dnsmichi, i open an issue with a summary of the things we talked about in the IRC Channel on Friday 09.07.10 about 3pm.
Description of the bug(?):
Scenario: 2 Boxes, one hosts Icinga (HP DL360 G5, 4GB RAM, Quadcore CPU, Debian Lenny 64bit), the other one hosts the MySQL (HP DL360 G3, 2GB RAM, 2 CPUs, Debian Lenny 64bit), both connected with Gigabit to the same Switch.
I upgrated from Icinga 1.0.1 and applied the fix upgrade-1.0.2-fix-object-relations.sql for the mysql as well as the mysql-upgrade-1.0.2.sql.
I use IDOUTILS with MySQL. When i restart or reload Icinga, the mysql process on the mysql-host uses 60 to 100% CPU for about one minute and the Icinga box has normal load.
In 1.0.1 the shell prompt was shown very fast after hitting return, now with 1.0.2 it takes about half a minute. The load of both boxes during that "restart period" is the same as in 1.0.1.
The most annoying thing is, that most of the interface checks (i use check_iftraffic.pl) that are running during this "restart period" are going UNKNOWN until the next checkrun.
I had this issue with RC1 and 1.0.1 when i ran Icinga and the Mysql DB on the same machine (the DL360 G5). But it was gone after moving the DB to a separate host. Now with 1.0.2 it is back. I have about 3645 objects in the icinga_objects table.

Member

icinga-migration commented Jul 19, 2010

Updated by mfriedrich on 2010-07-19 14:01:56 +00:00

so to speak the db is on a seperate host, but not ido2db itsself right? you could consider setting up

g5 icinga core with idomod+tcp socket
g3 ido2db+tcp socket with mysql rdbms

Maybe the altinity patch for extended hashslots was not that good for your setup. but i did not change that much in idoutils between those 2 releases except the database.. hmm.

Member

icinga-migration commented Jul 19, 2010

Updated by crsp on 2010-07-19 14:24:53 +00:00

You're so right, this is exactly my setup:

g5 icinga core with idomod+tcp socket
g3 ido2db+tcp socket with mysql rdbms

Member

icinga-migration commented Jul 27, 2010

Updated by mfriedrich on 2010-07-27 16:55:36 +00:00

hm, what are your config_output_options in idomod.cfg?

Member

icinga-migration commented Jul 28, 2010

Updated by crsp on 2010-07-28 08:34:25 +00:00

it's set to 1: config_output_options=1

Member

icinga-migration commented Aug 6, 2010

Updated by mfriedrich on 2010-08-06 18:27:26 +00:00

hm, show me

select * from icinga_instances;

select count(*) as cnt from icinga_objects where is_ative=0;

select count(*) as cnt from icinga_objects join icinga_instances on icinga_objects.instance_id=icinga_instances.instance_id where icinga_objects.is_ative=0 and icinga_instances.instance_name='YOUR CURRENT INSTANCE NAME IN IDOMOD.CFG';
Member

icinga-migration commented Aug 6, 2010

Updated by crsp on 2010-08-06 19:14:15 +00:00

here you are:

select * from icinga_instances;

------------~~------------------------------------~~
| instance_id | instance_name | instance_description |

------------~~------------------------------------~~
| 1 | default | |

------------~~------------------------------------~~
1 row in set (0.01 sec)

select count(*) as cnt from icinga_objects where is_ative=0;

ERROR 1054 (42S22): Unknown column 'is_ative' in 'where clause'

select count(*) as cnt from icinga_objects join icinga_instances on icinga_objects.instance_id=icinga_instances.instance_id where icinga_objects.is_ative=0 and icinga_instances.instance_name='YOUR CURRENT INSTANCE NAME IN IDOMOD.CFG';

mysql> select count(*) as cnt from icinga_objects join icinga_instances on icinga_objects.instance_id=icinga_instances.instance_id where icinga_objects.is_ative=0 and icinga_instances.instance_name='default';
ERROR 1054 (42S22): Unknown column 'icinga_objects.is_ative' in 'where clause'

Member

icinga-migration commented Aug 7, 2010

Updated by mfriedrich on 2010-08-07 10:17:47 +00:00

hach gottchen bisschen mitdenken :P

typo => is_ative == is_active

(show create table icinga_objects)

Member

icinga-migration commented Aug 7, 2010

Updated by crsp on 2010-08-07 15:06:14 +00:00

Mitdenken nu auch noch? :P
Ich vertraue immer dem Entwickler! :)

Aber hier:

mysql> select * from icinga_instances;
+-------------+---------------+----------------------+
| instance_id | instance_name | instance_description |
+-------------+---------------+----------------------+
|           1 | default       |                      |
+-------------+---------------+----------------------+
1 row in set (0.00 sec)

mysql>select count(*) as cnt from icinga_objects where is_active=0;

+-----+
| cnt |
+-----+
|  68 |
+-----+
1 row in set (0.01 sec)

mysql>select count(*) as cnt from icinga_objects join icinga_instances on icinga_objects.instance_id=icinga_instances.instance_id where icinga_objects.is_active=0 and icinga_instances.instance_name='default';

+-----+
| cnt |
+-----+
|  68 |
+-----+
1 row in set (0.02 sec)
Member

icinga-migration commented Sep 17, 2010

Updated by mfriedrich on 2010-09-17 15:32:50 +00:00

  • Status changed from New to Feedback

i can't reproduce that. can you recheck with 1.0.3 or the latest git master?

Member

icinga-migration commented Sep 30, 2010

Updated by crsp on 2010-09-30 17:22:28 +00:00

Updated today to the latest stuff from GIT. so far not too bad!
a restart produces about 40% load on the Mysql host. this is ways less than some time ago.
i test some more days, otherwise i think we are done.

mni tnx!!!

Member

icinga-migration commented Sep 30, 2010

Updated by mfriedrich on 2010-09-30 18:31:26 +00:00

ok very nice :)

Member

icinga-migration commented Jan 10, 2011

Updated by mfriedrich on 2011-01-10 17:20:24 +00:00

  • Status changed from Feedback to Closed

before i'll close the ticket - make sure you are using some way of ramdisk for pnp/perfdata. and put the mysql data stuff on an extended volume (like /var/lib), not on the same as /var/log and icinga.log reside.

but i think you'll find out yourself. closed for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment