Lograge in JSON mode in Rails causes double addition of log correlation information and breaks log parsing

[chall8908]

The current implementation of the Rails integration causes log correlation information to be added twice to the logger when using Lograge. This breaks log parsing in Datadog if Lograge is using the JSON formatter.

I'm currently attempting to work around this by setting Rails.configuration.log_tags = [] in initializers/datadog.rb.

As far as I can tell, this is likely going to be a problem in any Rails project that doesn't override the default logger when using Lograge (which doesn't modify the default logger) and using the JSON formatter.

I'm not sure if it's possible to simply delay modifying Rails.configuration.log_tags until after_initialize and still have everything function correctly. If nothing else, a mention in the README about this potential issue would save other people the head-scratching.

[marcotc]

👋 @chall8908, thanks for this issue report.
Could you provide the relevant Rails log configuration you use (Lograge configuration, any /config\..*log.*/ related Rails configuration) and the Rails version and Lograge you are running?
One possibility is that the problem is a mix of specific versions of Rails, Lograge and ddtrace combined.

[javierjulio]

Hi @marcotc 👋🏻 I'm noticing this as well. With Datadog v1.1.0 (we upgraded recently and directly from v0.54.2) and our Lograge integration, our logs have prepended the Datadog log text before the json format which does include the Datadog tracing info.

[dd.env=production dd.service=my-app dd.version=123 dd.trace_id=123 dd.span_id=123] {"method":"POST","path":"/","format":"html","controller":"SiteController","action":"index","status":200,"duration":90.02,"view":18.85,"db":25.61,"dd":{"trace_id":"123","span_id":"123","env":"production","service":"my-app","version":"123"},"ddsource":"ruby","time":728707.869930716,"params":{},"request_id":"123"}

Before replacing the info in the log shared above, the trace and span ids were a match in both text and json formats.

We are using lograge v0.12.0 (latest), ddtrace v1.1.0 (latest) and rails v6.1.6.

[javierjulio]

@marcotc we are actually overriding the default logger whereas with what @chall8908 wrote it seems they aren't but that is still resulting in the same output issue.

For Lograge config:

Rails.application.configure do
  config.lograge.enabled = true

  # Set the default log formatter but only if we have Lograge
  # enabled since we are using the Lograge JSON formatter.
  if config.lograge.enabled
    config.log_formatter = Lograge::Formatters::Json.new
    config.lograge.formatter = config.log_formatter
  end

  # Add our own custom data and Lograge will take care of merging this with
  # a base set. Do not merge `event.payload` (only merge explicit values)
  # as that contains a large object from ActionDispatch.
  config.lograge.custom_options = ->(event) do
    {
      time: event.time,
      params: event.payload[:params].except("controller", "action", "format")
    }.merge(event.payload[:custom_payload] || {})
  end

  # A hook to access controller methods so we can log common request info.
  config.lograge.custom_payload do |controller|
    {
      request_id: controller.request.uuid,
      user_id: controller.current_user.try(:id)
    }.compact
  end
end

For Rails logging:

  # Include generic and useful information about system operation, but avoid logging too much
  # information to avoid inadvertent exposure of personally identifiable information (PII).
  config.log_level = ENVied.LOG_LEVEL

  # Prepend all log lines with the following tags.
  # config.log_tags = [ :request_id ]

  # Log disallowed deprecations.
  config.active_support.disallowed_deprecation = :log

  # Tell Active Support which deprecation messages to disallow.
  config.active_support.disallowed_deprecation_warnings = []

  # Use default logging formatter so that PID and timestamp are not suppressed.
  # config.log_formatter = ::Logger::Formatter.new

  # Use a different logger for distributed setups.
  # require "syslog/logger"
  # config.logger = ActiveSupport::TaggedLogging.new(Syslog::Logger.new 'app-name')

  if ENVied.RAILS_LOG_TO_STDOUT
    config.logger = ActiveSupport::TaggedLogging.new(ActiveSupport::Logger.new(STDOUT))
  end

  # Log CSRF failures
  config.action_controller.log_warning_on_csrf_failure = true

[chall8908]

@marcotc sorry for leaving this for so long. We ended up pivoting off Lograge entirely and I got pulled away to other projects.

@javierjulio has the right of it, though. We were using Lograge more-or-less out-of-the-box. That particular project is on Rails 6 when I was testing it. Before deciding to ditch Lograge entirely, I'd pared down our configuration to just enabling Lograge and setting the JSON formatter. No other configuration.

Since Lograge doesn't change the Rails.logger by default, it usually remains as an ActiveSupport::TaggedLogging instance which triggers dd-trace to set log_tags which causes the problem.

I think swapping the add_logger call from before_initialize to after_initialize and actually checking if Lograge is enabled should fix the problem, but I'm not sure if that would cause downstream problems.

[eduardohertz]

Hey 👋

I was getting the same error as @javierjulio in my logs using the same stack. I'm not sure about the side effects but I've got a fix by disabling the log injection before rails initialization. I've added Datadog.configuration.tracing.log_injection = false in my config/environment.rb before Rails.application.initialize!, and the datadog tracing prefix was removed from my JSON logs.

Be advised: I'm sure about the side-effects of this solution 😄

EDIT: thank you @chall8908 for pointing out a way

[elia]

I can confirm the solution used by @eduardohertz, in our case we put the configuration in the config/environment/*.rb files, which are loaded at the right time.

Before that we were tried to remove the DD tagger manually with #Rails.configuration.log_tags.reject! { _1.source_location.first.match? 'lib/datadog/tracing' } but it didn't look pretty.

[javierjulio]

@chall8908 since you mentioned Lograge was removed, may I ask what you replaced it with and is that new thing still doing JSON formatting for logs? I'd be open to changing ours but I'm not familiar with another alternative.

@marcotc we haven't been able to upgrade from 0.54.2 due to this bug where the logs include both JSON and text output because of ddtrace. Is there any chance of it being resolved?

[chall8908]

@javierjulio we switched to Semantic Logger which does still do JSON logging (among other things).

[too-gee]

+1 to this issue. I have several teams that are running into this or will hit it in the near future and the docs don't offer a workaround.

[TonyCTHsu]

👋 @too-gee Thanks for reporting, I will try to reproduce it and address it.

[javierjulio]

@TonyCTHsu thank you!