Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

database: error instead of dying on monitor failure #301

Merged
merged 7 commits into from
Jul 23, 2019
Merged

Conversation

g-k
Copy link
Contributor

@g-k g-k commented Jul 19, 2019

Code changes:

  • change DB monitor to log errors instead of dying (only the csigpki signer uses the DB and it sets up and loads its config at init, so it's OK to briefly lose DB connectivity and we shouldn't crash the app)
  • factor out DB init to addDB in main
  • factor out CheckConnection func
  • add connect/monitor tests
  • add MonitorPollInterval to DB config
  • distinguish the heartbeat handler from LB heartbeat and teach it to check HSM connectivity (it finds the first signer config with a non-PEM private key, caches it, and uses it to check HSM connectivity returning 502 bad gateway when the connection fails to remove it from the health LBs cc @milescrabill @Micheletto re: this behavior)

Unit tests just check DB connected and HSM disconnected, but I functional tested that running app-hsm and hitting http://localhost:8001/__heartbeat__ returns

{"hsmAccessible": true}

r? @jvehent

TODO assuming this lands:

  • monitor logs for HSM and DB disconnects
  • add monitorpollinterval to autograph configs

@g-k g-k force-pushed the stability-fixes branch 3 times, most recently from 15641f2 to 48360eb Compare July 22, 2019 22:46
@coveralls
Copy link

coveralls commented Jul 22, 2019

Pull Request Test Coverage Report for Build 2981

  • 64 of 128 (50.0%) changed or added relevant lines in 4 files are covered.
  • 1 unchanged line in 1 file lost coverage.
  • Overall coverage decreased (-0.1%) to 70.86%

Changes Missing Coverage Covered Lines Changed/Added Lines %
database/connect.go 17 21 80.95%
handlers.go 34 43 79.07%
signer/signer.go 13 28 46.43%
main.go 0 36 0.0%
Files with Coverage Reduction New Missed Lines %
main.go 1 45.08%
Totals Coverage Status
Change from base Build 2955: -0.1%
Covered Lines: 3171
Relevant Lines: 4475

💛 - Coveralls

@g-k g-k requested a review from jvehent July 22, 2019 22:49
@g-k g-k marked this pull request as ready for review July 22, 2019 22:50
jvehent
jvehent previously approved these changes Jul 23, 2019
Copy link
Contributor

@jvehent jvehent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

r+, with some minor comments

database/connect_test.go Show resolved Hide resolved
database/connect_test.go Outdated Show resolved Hide resolved
handlers.go Outdated Show resolved Hide resolved
handlers_test.go Outdated Show resolved Hide resolved
handlers.go Show resolved Hide resolved
handlers.go Outdated Show resolved Hide resolved
@g-k g-k merged commit ca1009d into master Jul 23, 2019
@g-k g-k deleted the stability-fixes branch July 23, 2019 14:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants