[dev.icinga.com #886] ido2db forks if DB table not found

[icinga-migration]

This issue has been migrated from Redmine: https://dev.icinga.com/issues/886

Created by ralfk on 2010-10-13 12:05:19 +00:00

Assignee: mfriedrich
Status: Resolved (closed on 2010-10-20 15:37:35 +00:00)
Target Version: 1.3
Last Update: 2014-12-08 14:35:34 +00:00 (in Redmine)

Icinga Version: 1.10.0
OS Version: any

---

After deleting of all tables in icinga DB (in my case since I wanted to re-provision the tables again), the ido2db daemon was forking many processes and initiated many connects to the MySQL DB. Notice: I have not stopped ido2db or icinga daemons before I deleted the tables. In practical table deletion could also happen without intention.

Syslog Messages:

Oct 13 09:41:18 owl icinga: idomod: Successfully reconnected to data sink!  0 items lost, 17 queued items to flush.
Oct 13 09:41:18 owl icinga: idomod: Successfully flushed 17 queued items to data sink.
Oct 13 09:41:18 owl ido2db: Client connected, data available.
Oct 13 09:41:18 owl ido2db: Handling client connection...
Oct 13 09:41:18 owl ido2db: Successfully connected to mysql database
Oct 13 09:41:18 owl ido2db: Error: database query failed for 'SELECT instance_id FROM icinga_instances WHERE instance_name='mvtc1'' - '1146: Table 'icinga.icinga_instances' doesn't exist'
Oct 13 09:41:18 owl ido2db: Successfully disconnected from mysql database
Oct 13 09:41:18 owl ido2db: Error: Initial mysql database query failed! Please check mysql database configuration and schema!
Oct 13 09:41:18 owl ido2db: Exiting ...
Oct 13 09:41:18 owl ido2db: Handling client connection...
Oct 13 09:41:18 owl ido2db: Handling client connection...
Oct 13 09:41:18 owl ido2db: Handling client connection...
Oct 13 09:41:18 owl ido2db: Handling client connection...
Oct 13 09:41:18 owl ido2db: Successfully connected to mysql database
Oct 13 09:41:18 owl ido2db: Handling client connection...
Oct 13 09:41:18 owl ido2db: Handling client connection...
Oct 13 09:41:18 owl ido2db: Handling client connection...
Oct 13 09:41:18 owl ido2db: Handling client connection...
Oct 13 09:41:18 owl ido2db: Handling client connection...
Oct 13 09:41:18 owl ido2db: Handling client connection...
Oct 13 09:41:18 owl ido2db: Successfully connected to mysql database
Oct 13 09:41:18 owl ido2db: Successfully disconnected from mysql database
Oct 13 09:41:18 owl ido2db: Successfully shutdown... (PID=12293)
Oct 13 09:41:18 owl ido2db: Successfully connected to mysql database
Oct 13 09:41:18 owl ido2db: Handling client connection...
Oct 13 09:41:18 owl ido2db: Successfully connected to mysql database
Oct 13 09:41:18 owl ido2db: Successfully disconnected from mysql database
Oct 13 09:41:18 owl ido2db: Successfully shutdown... (PID=12292)

At the end the load was about 25 and other web applications were not able to connect to MySQL any more. Then I stopped both daemons with the init scripts, but ido2db was still flooding the system with a lot of new processes and DB connects. I had to killall the ido2db processes by hand. After that a ido2db start worked.

So, at least the ido2db daemon should not "crash" the system or the MySQL RDBMS when certain tables can't be found.

See also:

• https://dev.icinga.org/issues/773
• http://www.nagios-portal.org/wbb/index.php?page=Thread&threadID=19856

Changesets

2010-10-15 17:06:11 +00:00 by mfriedrich e748c9d

idoutils: fix disconnecting on errors misbheaving, not closing db and daemon #886

refs #886

[icinga-migration]

Updated by mfriedrich on 2010-10-15 17:05:23 +00:00

• Status changed from New to Assigned
• Assigned to set to mfriedrich
• Priority changed from Normal to High
• Target Version set to 1.3
• Done % changed from 0 to 50

i have an implementation of a fix in mfriedrich/core soon.

[icinga-migration]

Updated by ralfk on 2010-10-20 07:47:16 +00:00

Fix successfully tested against MySQL DB (by patching original 1.2.0 with changeset_re748c9dbd85c06485aa06a0ccc61314dedb4c404_rfadf228a1f983f3da505ed27afe8efe0de87748c.diff).

No forking ido2db processes any more if e.g. table icinga_instances is missing. Just these error messages appear in icinga.log:

[1287560041] idomod: Successfully reconnected to data sink!  0 items lost, 15 queued items to flush.
[1287560041] idomod: Successfully flushed 15 queued items to data sink.
[1287560047] idomod: Error writing to data sink!  Some output may get lost...
[1287560047] idomod: Please check remote ido2db log, database connection or SSL Parameters

Thanks.

[icinga-migration]

Updated by mfriedrich on 2010-10-20 12:18:10 +00:00

• Done % changed from 50 to 90

thanks for testing, mine runs fine too for the last days. i'll merge to master/r1.2.1 then.

[icinga-migration]

Updated by mfriedrich on 2010-10-20 15:37:35 +00:00

• Status changed from Assigned to Resolved
• Done % changed from 90 to 100

[icinga-migration]

Updated by mfriedrich on 2014-12-08 14:35:34 +00:00

• Project changed from 18 to Core, Classic UI, IDOUtils
• Category set to IDOUtils
• Icinga Version set to 1
• OS Version set to any