Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

on the test server, write ruby errors to cloudwatch and honeybadger #57662

Merged
merged 2 commits into from Apr 10, 2024

Conversation

davidsbailey
Copy link
Member

@davidsbailey davidsbailey commented Mar 29, 2024

today it is difficult to debug server errors on the test machine. this is especially bad when working with Active Job, because there is no easy way to surface these errors in the UI. Logs are currently written to dashboard/log/test.log, in rails format, which does not include a timestamp:

Started GET "/print_certificates/eyJuYW1lIjoiUHVzaHRp" for 71.212.102.136 at 2024-03-29 19:10:58 +0000
Processing by PrintCertificatesController#show as HTML
  Parameters: {"encoded_params"=>"eyJuYW1lIjoiUHVzaHRp"}
Completed 500 Internal Server Error in 2ms (Allocations: 911)
  
JSON::ParserError (859: unexpected token at '{"name":"Pushti'):
  
app/controllers/print_certificates_controller.rb:15:in `show'
app/controllers/application_controller.rb:186:in `block in with_locale'
app/controllers/application_controller.rb:185:in `with_locale'
** [Honeybadger] Initializing Honeybadger Error Tracker for Ruby. Ship it! version=4.12.1 framework=rails level=1 pid=1864053
** [Honeybadger] Initializing Honeybadger Error Tracker for Ruby. Ship it! version=4.12.1 framework=rails level=1 pid=1864053
** [Honeybadger] Development mode is enabled. Data will not be reported until you deploy your app. level=2 pid=1864053
** [Honeybadger] Reporting error id=5829e667-c308-47e3-a912-79bc0357b015 level=1 pid=1767892
** [Honeybadger] Success ⚡ Development mode is enabled; this error will be reported if it occurs after you deploy your app. id=5829e667-c308-47e3-a912-79bc0357b015 level=1 pid=1767892
** [Honeybadger] Initializing Honeybadger Error Tracker for Ruby. Ship it! version=4.12.1 framework=rails level=1 pid=1864348
** [Honeybadger] Development mode is enabled. Data will not be reported until you deploy your app. level=2 pid=1864348

This config change makes the logs show up in cloudwatch and also includes a timestamp in /var/log/syslog:

Mar 29 20:33:28 test dashboard[1942529]: @cee: {"method":"GET","path":"/print_certificates/eyJuYW1lIjoiUHVzaHRp","format":"html","controller":"PrintCertificatesController","action":"show","status":500,"error":"JSON::ParserError: 859: unexpected token at '{\"name\":\"Pushti'","duration":6.85,"view":0.0}
Mar 29 20:33:28 test dashboard[1942529]: @cee: {"method":"GET","path":"/print_certificates/eyJuYW1lIjoiUHVzaHRp","format":"html","controller":null,"action":null,"status":500,"error":"JSON::ParserError: 859: unexpected token at '{\"name\":\"Pushti'","duration":0.01}
Mar 29 20:33:28 test dashboard[1942282]: Rendered ActiveModel::Serializer::CollectionSerializer with Array (29.58ms)
Mar 29 20:33:28 test dashboard[1942282]: @cee: {"method":"GET","path":"/api/v1/test_logs/test/test/since/1711741146","format":"json","controller":"Api::V1::TestLogsController","action":"get_logs_since","status":200,"duration":1604.8,"view":33.48}
Mar 29 20:33:29 test dashboard[1942529]: ** [Honeybadger] Reporting error id=beb69396-87b5-40c8-a8e8-389518878dd9 level=1 pid=1942529
Mar 29 20:33:29 test dashboard[1942529]: ** [Honeybadger] Success ⚡ Development mode is enabled; this error will be reported if it occurs after you deploy your app. id=beb69396-87b5-40c8-a8e8-389518878dd9 level=1 pid=1942529

f5c0597 also makes the error details show up in honey badger.

Testing story

patched onto the test machine, and verified that new ruby errors appear in:

@davidsbailey davidsbailey changed the title on the test server, write ruby errors to syslog and cloudwatch on the test server, write ruby errors to cloudwatch and honeybadger Mar 29, 2024
Comment on lines +14 to +15
test:
report_data: true
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davidsbailey davidsbailey marked this pull request as ready for review March 29, 2024 22:51
Comment on lines +78 to +87
if CDO.running_web_application?
# Use default logging formatter so that PID and timestamp are not suppressed.
config.log_formatter = Logger::Formatter.new

# Log condensed lines to syslog for centralized logging.
config.lograge.enabled = true
config.lograge.formatter = Lograge::Formatters::Cee.new
require 'syslog/logger'
config.logger = Syslog::Logger.new 'dashboard', Syslog::LOG_LOCAL0
end
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

similar logging config to other environments, but limited to running web app to make sure it does not get set in unit tests.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 in particular, very similar to the config for production.rb

Copy link
Contributor

@snickell snickell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me - generally like leaning into honeybadger.

Comment on lines +78 to +87
if CDO.running_web_application?
# Use default logging formatter so that PID and timestamp are not suppressed.
config.log_formatter = Logger::Formatter.new

# Log condensed lines to syslog for centralized logging.
config.lograge.enabled = true
config.lograge.formatter = Lograge::Formatters::Cee.new
require 'syslog/logger'
config.logger = Syslog::Logger.new 'dashboard', Syslog::LOG_LOCAL0
end
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 in particular, very similar to the config for production.rb

@davidsbailey davidsbailey merged commit 4a30734 into staging Apr 10, 2024
1 of 2 checks passed
@davidsbailey davidsbailey deleted the write-errors-to-test-syslog branch April 10, 2024 18:01
@davidsbailey davidsbailey mentioned this pull request Apr 12, 2024
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants