Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CLI Task "jobs run" dies after database connection loss #188

Closed
acp-jsz opened this issue Jun 28, 2023 · 0 comments · Fixed by Icinga/icingadb-web#859
Closed

CLI Task "jobs run" dies after database connection loss #188

acp-jsz opened this issue Jun 28, 2023 · 0 comments · Fixed by Icinga/icingadb-web#859
Labels
bug Something isn't working

Comments

@acp-jsz
Copy link

acp-jsz commented Jun 28, 2023

Describe the bug

We have recently upgraded the module from version 1.1.2 to version 1.2.1.
Since this update the Service (icinga-x509.service) dies everytime, after the database connection got lost
e.g. when the database got restartet.
Also the icinga-php-library got updated from v0.11.0 to v0.12.0

systemd[1]: Started Icinga Certificate Monitoring Module Jobs Runner.
icingacli[10105]: PHP Fatal error:  Uncaught PDOException: SQLSTATE[70100]: <<Unknown error>>: 1927 Connection was killed in /usr/share/icinga-php/ipl/vendor/ipl/sql/src/Connection.php:401
icingacli[10105]: Stack trace:
icingacli[10105]: #0 /usr/share/icinga-php/ipl/vendor/ipl/sql/src/Connection.php(401): PDO->prepare('SELECT host.nam...')
icingacli[10105]: #1 /usr/share/icinga-php/ipl/vendor/ipl/sql/src/Connection.php(418): ipl\Sql\Connection->prepexec('SELECT host.nam...', Array)
icingacli[10105]: #2 /usr/share/icingaweb2/modules/icingadb/library/Icingadb/ProvidedHook/X509/Sni.php(45): ipl\Sql\Connection->select(Object(ipl\Sql\Select))
icingacli[10105]: #3 /usr/share/icingaweb2/modules/x509/library/X509/Hook/SniHook.php(32): Icinga\Module\Icingadb\ProvidedHook\X509\Sni->getHosts()
icingacli[10105]: #4 /usr/share/icingaweb2/modules/x509/application/clicommands/JobsCommand.php(128): Icinga\Module\X509\Hook\SniHook::getAll()
icingacli[10105]: #5 /usr/share/icingaweb2/modules/x509/application/clicommands/JobsCommand.php(57): Icinga\Module\X509\Clicommands\JobsCommand->fetchJobs()
icingacli[10105]: #6 /usr/share/icinga-php/vendor/vendor/react/event-loop/src/Timer/Timers.php(101): Icinga\M in /usr/share/icinga-php/ipl/vendor/ipl/sql/src/Connection.php on line 401
icingacli[10105]: Fatal error: Uncaught PDOException: SQLSTATE[70100]: <<Unknown error>>: 1927 Connection was killed in /usr/share/icinga-php/ipl/vendor/ipl/sql/src/Connection.php:401
icingacli[10105]: Stack trace:
icingacli[10105]: #0 /usr/share/icinga-php/ipl/vendor/ipl/sql/src/Connection.php(401): PDO->prepare('SELECT host.nam...')
icingacli[10105]: #1 /usr/share/icinga-php/ipl/vendor/ipl/sql/src/Connection.php(418): ipl\Sql\Connection->prepexec('SELECT host.nam...', Array)
icingacli[10105]: #2 /usr/share/icingaweb2/modules/icingadb/library/Icingadb/ProvidedHook/X509/Sni.php(45): ipl\Sql\Connection->select(Object(ipl\Sql\Select))
icingacli[10105]: #3 /usr/share/icingaweb2/modules/x509/library/X509/Hook/SniHook.php(32): Icinga\Module\Icingadb\ProvidedHook\X509\Sni->getHosts()
icingacli[10105]: #4 /usr/share/icingaweb2/modules/x509/application/clicommands/JobsCommand.php(128): Icinga\Module\X509\Hook\SniHook::getAll()
icingacli[10105]: #5 /usr/share/icingaweb2/modules/x509/application/clicommands/JobsCommand.php(57): Icinga\Module\X509\Clicommands\JobsCommand->fetchJobs()
icingacli[10105]: #6 /usr/share/icinga-php/vendor/vendor/react/event-loop/src/Timer/Timers.php(101): Icinga\M in /usr/share/icinga-php/ipl/vendor/ipl/sql/src/Connection.php on line 401
systemd[1]: icinga-x509.service: Main process exited, code=exited, status=255/EXCEPTION
systemd[1]: icinga-x509.service: Failed with result 'exit-code'.

To Reproduce

  1. Start the service:
    systemctl start icinga-x509.service
  2. Restart the Database (MySQL, on same host)
    systemctl restart mysql
  3. Wait for next actions of the jobs run-Task (not sure when it next time occures)
    (on latest test it occured five minutes after the database restart)
  4. Daemon dies

Expected behavior

The jobs run-Task should try to reconnect to the database and continue to work.

Screenshots

No Screenshots available. See Stacktrace above.

Your Environment

Include as many relevant details about the environment you experienced the problem in

  • Icinga Web 2 version and modules (System - About):
    Icinga Web 2 Version 2.11.4

  • Web browser used:
    No Webbrowser used, CLI related

  • Icinga 2 version used (icinga2 --version):
    r2.13.7-1

  • PHP version used (php --version):
    7.3.31-1~deb10u4

  • Server operating system and version:
    Debian 10 (Buster)

Additional context

We do have three Jobs configured. Those already changed to the new schedule-config and configured to run daily.

  • 4:30
  • 5:43
  • 7:45

A regular restart task for the database is actually running at 6:30, every day.

@yhabteab yhabteab added the bug Something isn't working label Jul 7, 2023
@yhabteab yhabteab added this to the 1.2.2 milestone Jul 7, 2023
@raviks789 raviks789 assigned yhabteab and unassigned yhabteab Sep 1, 2023
@nilmerg nilmerg linked a pull request Sep 12, 2023 that will close this issue
@nilmerg nilmerg removed this from the 1.3.0 milestone Sep 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants