Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dev.icinga.com #11103] Problem with hostgroup_members table cleanup #3905

Closed
icinga-migration opened this issue Feb 6, 2016 · 15 comments
Closed
Labels
Milestone

Comments

@icinga-migration
Copy link
Member

@icinga-migration icinga-migration commented Feb 6, 2016

This issue has been migrated from Redmine: https://dev.icinga.com/issues/11103

Created by wstiern on 2016-02-06 03:11:36 +00:00

Assignee: mfriedrich
Status: Resolved (closed on 2016-02-08 10:10:04 +00:00)
Target Version: 2.4.2
Last Update: 2016-02-23 09:58:48 +00:00 (in Redmine)

Icinga Version: 2.4.1
Backport?: Already backported
Include in Changelog: 1

At some point this series of errors is thrown in the icinga2.log:

  state_type = '1',  status_update_time = FROM_UNIXTIME(1454727237) WHERE service_object_id = 4858;INSERT INTO icinga_statehistory (check_source, current_check_attempt, endpoint_object_id, instance_id, last_hard_state, last_state, long_output, max_check_attempts, object_id, output, state, state_change, state_time, state_time_usec, state_type) VALUES ('icinga.pulse.prod', '1', 161, 1, '3', '3', '', '3', 4858, '', '3', '1', FROM_UNIXTIME(1454727237), '987210', '1');INSERT INTO icinga_logentries (endpoint_object_id, entry_time, entry_time_usec, instance_id, logentry_data, logentry_time, logentry_type, object_id) VALUES (161, FROM_UNIXTIME(1454727238), '19208', 1, 'SERVICE ALERT: lga-rtb14;rtb-node-dataservices-exceptions-0;UNKNOWN;HARD;1;', FROM_UNIXTIME(1454727238), '16384', 4697);UPDATE icinga_servicestatus SET acknowledgement_type = '0',  active_checks_enabled = '1',  check_command = 'graphite',  check_source = 'icinga.pulse.prod',  check_type = '0',  current_check_attempt = '1',  current_notification_number = '0',  current_state = '3',  endpoint_object_id = 161,  event_handler = '',  event_handler_enabled = '1',  execution_time = '15.483777046203613',  flap_detection_enabled = '0',  has_been_checked = '1',  instance_id = 1,  is_flapping = '0',  is_reachable = '1',  last_check = FROM_UNIXTIME(1454727238),  last_hard_state = '3',  last_hard_state_change = FROM_UNIXTIME(1454727238),  last_state_change = FROM_UNIXTIME(1454727207),  latency = '0',  long_output = '',  max_check_attempts = '3',  next_check = FROM_UNIXTIME(1454727822),  normal_check_interval = '10',  notifications_enabled = '1',  original_attributes = 'null',  output = '',  passive_checks_enabled = '1',  percent_state_change = '0',  perfdata = '',  problem_has_been_acknowledged = '0',  process_performance_data = '1',  retry_check_interval = '2.5',  scheduled_downtime_depth = '0',  service_object_id = 4697,  should_be_scheduled = '1',  state_type = '1',  status_update_time = FROM_UNIXTIME(1454727238) WHERE service_object_id = 4697;INSERT INTO icinga_statehistory (check_source, current_check_attempt, endpoint_object_id, instance_id, last_hard_state, last_state, long_output, max_check_attempts, object_id, output, state, state_change, state_time, state_time_usec, state_type) VALUES ('icinga.pulse.prod', '1', 161, 1, '3', '3', '', '3', 4697, '', '3', '1', FROM_UNIXTIME(1454727238), '18850', '1')"
[2016-02-05 21:54:01 -0500] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!
[2016-02-05 21:54:01 -0500] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.14.0')

When this happens, the Icingaweb2 UI loses contact with the daemon ("Backend Icinga is not available") and doesn't recover until I stop icinga2, nuke /var/lib/icinga2/icinga2.state, and start icinga2 again. I've wiped and rebuilt both IDO and Icingaweb DBs many times. Please let me know what other information I can provide.

Attachments

  • icinga2.error wstiern - 2016-02-06 03:12:51 +00:00 - Excerpt from error log during DB failure.

Changesets

2016-02-08 10:05:27 +00:00 by mfriedrich 99dbb24

DB IDO: Fix problem with hostgroup_members cleanup

fixes #11103

2016-02-22 19:23:18 +00:00 by mfriedrich 5b30c9e

DB IDO: PArtially revert change for *group_members

Otherwise the index breaks.

refs #11103

2016-02-23 08:45:10 +00:00 by mfriedrich 2b64364

DB IDO: Fix problem with hostgroup_members cleanup

fixes #11103

2016-02-23 08:45:10 +00:00 by mfriedrich 2ec36d2

DB IDO: PArtially revert change for *group_members

Otherwise the index breaks.

refs #11103

Relations:

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Feb 6, 2016

Updated by wstiern on 2016-02-06 03:13:02 +00:00

  • File added icinga2.error

Totally botched the error text. Here's a file.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Feb 6, 2016

Updated by tobiasvdk on 2016-02-06 20:22:22 +00:00

  • Relates set to 11017
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Feb 8, 2016

Updated by mfriedrich on 2016-02-08 09:50:23 +00:00

  • Relates deleted 11017
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Feb 8, 2016

Updated by mfriedrich on 2016-02-08 10:00:47 +00:00

  • Relates set to 10855
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Feb 8, 2016

Updated by mfriedrich on 2016-02-08 10:01:56 +00:00

  • Subject changed from State File Corruption to Problem with hostgroup_members table cleanup
  • Category set to DB IDO

The delete/insert queries are not put into a query group required by the changes in #8714 and #10855.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Feb 8, 2016

Updated by mfriedrich on 2016-02-08 10:02:04 +00:00

  • Relates set to 8714
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Feb 8, 2016

Updated by mfriedrich on 2016-02-08 10:06:03 +00:00

  • Status changed from New to Assigned
  • Assigned to set to mfriedrich
  • Target Version set to 2.4.2
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Feb 8, 2016

Updated by mfriedrich on 2016-02-08 10:07:23 +00:00

Please re-test using the latest snapshot packages (icinga2 --version == v2.4.1-178-g99dbb24).

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Feb 8, 2016

Updated by mfriedrich on 2016-02-08 10:10:04 +00:00

  • Status changed from Assigned to Resolved
  • Done % changed from 0 to 100

Applied in changeset 99dbb24.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Feb 8, 2016

Updated by wstiern on 2016-02-08 15:13:01 +00:00

dnsmichi wrote:

Please re-test using the latest snapshot packages (icinga2 --version == v2.4.1-178-g99dbb24).

Pardon my ignorance here, but I can't find a RPM that matches that version in the repo. There do appear to be daily snapshots, should I grab one from today?

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Feb 8, 2016

Updated by mfriedrich on 2016-02-08 15:23:16 +00:00

Which distribution are you on?

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Feb 8, 2016

Updated by wstiern on 2016-02-08 15:54:36 +00:00

dnsmichi wrote:

Which distribution are you on?

CentOS 7. I've just added the snapshot branch of the filesystem as another Yum repo. If there's a specific datestamped version I should use let me know, otherwise I'll test with whatever's current. Thanks for your help! :)

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Feb 9, 2016

Updated by mfriedrich on 2016-02-09 14:58:14 +00:00

el7 packages are built on each git commit so you should be fine using the current revision. Once you've installed the package, run icinga2 --version and verify its working (post your results here then).

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Feb 22, 2016

Updated by mfriedrich on 2016-02-22 19:22:43 +00:00

  • Relates set to 11133
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Feb 23, 2016

Updated by gbeutner on 2016-02-23 09:58:49 +00:00

  • Backport? changed from Not yet backported to Already backported
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
You can’t perform that action at this time.