Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dev.icinga.com #10829] IDO breaks when writing to icinga_programstatus with latest snapshots #3774

Closed
icinga-migration opened this issue Dec 11, 2015 · 9 comments

Comments

@icinga-migration
Copy link
Member

@icinga-migration icinga-migration commented Dec 11, 2015

This issue has been migrated from Redmine: https://dev.icinga.com/issues/10829

Created by tgelf on 2015-12-11 16:25:25 +00:00

Assignee: mfriedrich
Status: Resolved (closed on 2015-12-15 11:04:50 +00:00)
Target Version: 2.4.2
Last Update: 2016-02-23 09:58:31 +00:00 (in Redmine)

Icinga Version: 2.4.1
Backport?: Already backported
Include in Changelog: 0

Hi,

I'm seeing this with the latest snapshots:

critical/TcpSocket: Invalid socket: Connection refused
critical/IdoMysqlConnection: Error "Duplicate entry '1' for key 'instance_id'" when executing query "INSERT INTO icinga_programstatus (active_host_checks_enabled, active_service_checks_enabled, daemon_mode, endpoint_name, event_handlers_enabled, flap_detection_enabled, instance_id, is_currently_running, last_command_check, notifications_enabled, passive_host_checks_enabled, passive_service_checks_enabled, process_id, process_performance_data, program_start_time, program_version, status_update_time) VALUES ('1', '1', '1', 'master1', '1', '1', 1, '1', FROM_UNIXTIME(1449850710), '1', '1', '1', '19928', '1', FROM_UNIXTIME(1449847581), 'v2.4.1-64-gc5b13ff', FROM_UNIXTIME(1449850710));DELETE FROM icinga_programstatus WHERE instance_id = 1;INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_scheduled_services', '6');INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_services', '6');INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_hosts', '4');INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_scheduled_hosts', '4');DELETE FROM icinga_runtimevariables WHERE instance_id = 1;COMMIT;BEGIN"
critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!

Could this be related to recent insert/update fixes?

Cheers,
Thomas

Attachments

Changesets

2015-12-15 10:58:50 +00:00 by mfriedrich da3d210

Fix priority ordering for IDO queries

fixes #10829
refs #8714

2015-12-16 12:57:29 +00:00 by mfriedrich 6387bec

Fix duplicate key violation for downtimes

refs #10829

2016-02-23 08:09:06 +00:00 by mfriedrich 2bc1d32

Fix priority ordering for IDO queries

fixes #10829
refs #8714

2016-02-23 08:15:48 +00:00 by mfriedrich 187a61d

Fix duplicate key violation for downtimes

refs #10829

Relations:

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by tgelf on 2015-12-11 16:46:01 +00:00

  • Priority changed from High to Normal

Lowering prio, does not happen all the times... strange. Any ideas? Could it happen after unclean disconnects or similar?

Cheers,
Thomas

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 14, 2015

Updated by tgelf on 2015-12-14 09:08:59 +00:00

Well, doesn't happen all the times - but again and again. I think I found the reason for this:

critical/IdoMysqlConnection: Error "Duplicate entry '1' for key 'instance_id'" when executing query "
INSERT INTO icinga_programstatus (active_host_checks_enabled, ...) VALUES
('1', '1', '1', 'master1', '1', '1', 1, '1', FROM_UNIXTIME(1449926784), '1', '1', '1', '31722', '1',
FROM_UNIXTIME(1449926494), 'v2.4.1-69-g8055f05', FROM_UNIXTIME(1449926784));
DELETE FROM icinga_programstatus WHERE instance_id = 1;
INSERT INTO icinga_runtimevariables..."

As we can see it inserts new programstatus before deleting the old one. This happens in the very same query execution.

Best,
Thomas

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 14, 2015

Updated by mfriedrich on 2015-12-14 09:22:08 +00:00

  • Relates set to 8714
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 14, 2015

Updated by mfriedrich on 2015-12-14 17:04:14 +00:00

  • Priority changed from Normal to High

    [2015-12-14 18:03:15 +0100] information/ConfigItem: Instantiated 1 Downtime.
    [2015-12-14 18:03:15 +0100] information/ConfigItem: Activated all objects.
    [2015-12-14 18:03:15 +0100] information/ConfigItem: Triggering Start signal for config items
    [2015-12-14 18:03:15 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/modified-attributes.conf
    [2015-12-14 18:03:15 +0100] information/ConfigItem: Activated all objects.
    [2015-12-14 18:03:16 +0100] critical/IdoPgsqlConnection: Error "ERROR: duplicate key value violates unique constraint "uq_scheduleddowntime"
    DETAIL: Key (instance_id, object_id, entry_time, internal_downtime_id)=(1, 178, 2015-12-14 18:03:15+01, 39) already exists.
    " when executing query "INSERT INTO icinga_scheduleddowntime (author_name, comment_data, downtime_type, duration, endpoint_object_id, entry_time, instance_id, internal_downtime_id, is_fixed, is_in_effect, name, object_id, scheduled_end_time, scheduled_start_time, was_started) VALUES (E'icingaadmin', E'Scheduled downtime for backup', E'1', E'0', 1, TO_TIMESTAMP(1450112595), 1, E'39', E'1', E'0', E'mbmif.int.netways.de!load!mbmif.int.netways.de-1450112595-0', 178, TO_TIMESTAMP(1450144800), TO_TIMESTAMP(1450141200), E'0')"
    [2015-12-14 18:03:16 +0100] warning/IdoPgsqlConnection: Exception during database operation: Verify that your database is operational!
    [2015-12-14 18:03:20 +0100] information/Notification: Sending notification 'mbmif.int.netways.de!procs!mail-icingaadmin' for user 'icingaadmin2'
    [2015-12-14 18:03:20 +0100] information/Notification: Sending notification 'mbmif.int.netways.de!procs!mail-icingaadmin' for user 'icingaadmin'
    [2015-12-14 18:03:20 +0100] information/Notification: Sending notification 'mbmif.int.netways.de!disk /!mail-icingaadmin' for user 'icingaadmin2'
    [2015-12-14 18:03:20 +0100] information/Notification: Completed sending notification 'mbmif.int.netways.de!procs!mail-icingaadmin' for checkable 'mbmif.int.netways.de!procs'
    [2015-12-14 18:03:20 +0100] information/Notification: Sending notification 'mbmif.int.netways.de!disk /!mail-icingaadmin' for user 'icingaadmin'
    [2015-12-14 18:03:20 +0100] information/Notification: Completed sending notification 'mbmif.int.netways.de!procs!mail-icingaadmin' for checkable 'mbmif.int.netways.de!procs'
    [2015-12-14 18:03:20 +0100] information/Notification: Completed sending notification 'mbmif.int.netways.de!disk /!mail-icingaadmin' for checkable 'mbmif.int.netways.de!disk /'
    [2015-12-14 18:03:20 +0100] information/Notification: Sending notification 'mbmif.int.netways.de!disk!mail-icingaadmin' for user 'icingaadmin2'
    [2015-12-14 18:03:20 +0100] information/Notification: Sending notification 'mbmif.int.netways.de!disk!mail-icingaadmin' for user 'icingaadmin'
    [2015-12-14 18:03:20 +0100] information/Notification: Sending notification 'mbmif.int.netways.de!ido-mysql!mail-icingaadmin' for user 'icingaadmin2'
    [2015-12-14 18:03:20 +0100] information/Notification: Sending notification 'mbmif.int.netways.de!ido-mysql!mail-icingaadmin' for user 'icingaadmin'
    [2015-12-14 18:03:20 +0100] information/Notification: Completed sending notification 'mbmif.int.netways.de!ido-mysql!mail-icingaadmin' for checkable 'mbmif.int.netways.de!ido-mysql'
    [2015-12-14 18:03:20 +0100] information/Notification: Completed sending notification 'mbmif.int.netways.de!disk /!mail-icingaadmin' for checkable 'mbmif.int.netways.de!disk /'
    [2015-12-14 18:03:20 +0100] information/Notification: Completed sending notification 'mbmif.int.netways.de!disk!mail-icingaadmin' for checkable 'mbmif.int.netways.de!disk'
    [2015-12-14 18:03:23 +0100] information/Notification: Completed sending notification 'mbmif.int.netways.de!disk!mail-icingaadmin' for checkable 'mbmif.int.netways.de!disk'
    [2015-12-14 18:03:23 +0100] information/Notification: Completed sending notification 'mbmif.int.netways.de!ido-mysql!mail-icingaadmin' for checkable 'mbmif.int.netways.de!ido-mysql'
    [2015-12-14 18:03:25 +0100] information/IdoPgsqlConnection: pgSQL IDO instance id: 1 (schema version: '1.14.0')
    [2015-12-14 18:03:35 +0100] critical/IdoPgsqlConnection: Error "ERROR: duplicate key value violates unique constraint "uq_programstatus"
    DETAIL: Key (instance_id)=(1) already exists.
    " when executing query "INSERT INTO icinga_programstatus (active_host_checks_enabled, active_service_checks_enabled, daemon_mode, endpoint_name, event_handlers_enabled, flap_detection_enabled, instance_id, is_currently_running, last_command_check, notifications_enabled, passive_host_checks_enabled, passive_service_checks_enabled, process_id, process_performance_data, program_start_time, program_version, status_update_time) VALUES (E'1', E'1', E'1', E'mbmif.int.netways.de', E'1', E'1', 1, E'1', TO_TIMESTAMP(1450112615), E'1', E'1', E'1', E'17506', E'1', TO_TIMESTAMP(1450112594), E'v2.4.1-75-gf297ccd', TO_TIMESTAMP(1450112615))"
    [2015-12-14 18:03:35 +0100] warning/IdoPgsqlConnection: Exception during database operation: Verify that your database is operational!
    [2015-12-14 18:03:35 +0100] information/IdoPgsqlConnection: pgSQL IDO instance id: 1 (schema version: '1.14.0')
    [2015-12-14 18:03:45 +0100] critical/IdoPgsqlConnection: Error "ERROR: duplicate key value violates unique constraint "uq_programstatus"
    DETAIL: Key (instance_id)=(1) already exists.
    " when executing query "INSERT INTO icinga_programstatus (active_host_checks_enabled, active_service_checks_enabled, daemon_mode, endpoint_name, event_handlers_enabled, flap_detection_enabled, instance_id, is_currently_running, last_command_check, notifications_enabled, passive_host_checks_enabled, passive_service_checks_enabled, process_id, process_performance_data, program_start_time, program_version, status_update_time) VALUES (E'1', E'1', E'1', E'mbmif.int.netways.de', E'1', E'1', 1, E'1', TO_TIMESTAMP(1450112625), E'1', E'1', E'1', E'17506', E'1', TO_TIMESTAMP(1450112594), E'v2.4.1-75-gf297ccd', TO_TIMESTAMP(1450112625))"
    [2015-12-14 18:03:45 +0100] warning/IdoPgsqlConnection: Exception during database operation: Verify that your database is operational!

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 14, 2015

Updated by mfriedrich on 2015-12-14 17:04:44 +00:00

  • Target Version set to 2.4.2
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 14, 2015

Updated by mfriedrich on 2015-12-14 17:06:24 +00:00

  • File added debug.log.gz
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 15, 2015

Updated by mfriedrich on 2015-12-15 10:18:57 +00:00

  • Status changed from New to Assigned
  • Assigned to set to mfriedrich
  • Include in Changelog changed from 1 to 0
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 15, 2015

Updated by mfriedrich on 2015-12-15 11:04:50 +00:00

  • Status changed from Assigned to Resolved
  • Done % changed from 0 to 100

Applied in changeset da3d210.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Feb 23, 2016

Updated by gbeutner on 2016-02-23 09:58:31 +00:00

  • Backport? changed from Not yet backported to Already backported
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
You can’t perform that action at this time.