Money Configuration

Money uses the Typesafe Config Library for configuring the system. Money follows the recommendations set forth by the config library; namely it provides "sane" default values for everything that can be configured, and allows you to override the default configuration by providing a file named application.conf on the classpath of your application.

Core configuration

• application-name - set this to the name you want reported for your application. It is best not to use spaces in the application name. This value is reported on each Span as an app-name Note. If you see app-name=unknown, it indicates that you have not configured the application-name.
• override-environment-variable - Defaults to MONEY_ENV. Money provides a built in facility that allows you to have different configuration files for different environments. This value is set to a string that is used to lookup the application configuration file. For example, if you had a test environment and a prod environment, you would have an application.test.conf and application.prod.conf files both available on your classpath. When the environment variable MONEY_ENV=test, then Money will load with the application.test.conf file. When the environment variable is MONEY_ENV=prod, then Money will use the overrides in the application.prod.conf file. More information can be found below.
• enabled - true or false, depending on whether you want money turned on or not. If this is set to false, then all requests through the Money system will be no-op.
• span-timeout - this is an important configuration flag, that specifies how long a single Span is allowed to be open before it is shut down. Spans that timeout are not emitted; but a log entry will be made indicating that the span timed out. Also, a JMX property will be updated indicating a timed out span. This value should be set to the shortest acceptable time duration; otherwise, under heavy volume you can experience increased memory pressure due to lots of spans hanging out waiting to timeout.
• stopped-span-timeout - this is how long a span will remain open to accept additional notes after it has been Stopped. We keep spans open for a short time to allow fire-and-forget type of requests to complete and log data. We also keep spans open for Scala Futures, which can stop a span multiple times depending upon how they are used. This should also be kept to the smallest possible value. The default is 100 milliseconds, but it should be only 5 or 10 milliseconds.
• mdc.enabled - true or false, indicates whether or not you want to use the SLF4J MDC value. If enabled, then you can include the current Trace Context in every log line. This is very helpful when correlating log entries with span data.
• akka.loglevel - the log level for the Money Akka system. This is by default set to INFO, and it must be at least WARN in order to get Span Data. The default is usually acceptable.
• emitters - emitters will allow you to configure the individual emitters that will be handling trace data. They are included in their own section below.

Example Configuration File:

money {
  application-name = "my-app"
  enabled = true
  span-timeout = 5 seconds
  stopped-span-timeout = 5 ms

  mdc {
    enabled = true
  }

  akka {
    loglevel = INFO
    log-config-on-start = off
  }
}

Emitters

Emitters are the method by which Money will send (or emit) data. Money comes with a few out-of-the-box Emitters that you can use.

The Emitter Configuration is an Array of emitters, so you can have 0 or 100 different emitters

Log Emitter

The log emitter is our standard setup that we use by default. This allows us to get data into Heka, LogStash, or Splunk very easily. There is no configuration needed other than enabling it in the configuration file.

All logging in Money is performed via SLF4J. You need to configure your application as appropriate to log data, we typically use a non-blocking logger like Logback.

Configuration Example

money {
  emitter {
    emitters = [
      {
        name = "log-emitter"
        class-name = "com.comcast.money.emitters.LogEmitter"
        subscriptions = [Trace]
        configuration = {
          emitter = "com.comcast.money.emitters.LogEmitter"
        }
      }
    ]
  }
}

Span Metrics Emitter

The Span Metrics Emitter calculates aggregate data for each unique span-name in the application. Even in large applications, the total number of unique span names (operations) is below 1000. The Span Metrics Emitter exposes the metrics it collects via JMX. All metrics are captured using Dropwizard Metrics.

The Span Metrics Emitter calculates the following aggregates

• latency histogram - calculates the distribution of latency (duration) of each span name - See Histogram
• error rate - calculates the rate of errors for a particular span - See Meters

Configuration Example

money {
  emitter {
    emitters = [
      {
        name = "span-metrics-emitter"
        class-name = "com.comcast.money.metrics.SpanMetricsCollector"
        subscriptions = [Trace]
        configuration = {
        }
      }
    ]
  }
}

Graphite Emitter

The Graphite Emitter supports sending metrics to a Graphite server.

Configuration Example

         {
             name = "graphite-emitter"
             class-name = "com.comcast.money.emitters.GraphiteEmitter"
             subscriptions = [Trace, Metric]
             configuration = {
                 app-name = "inventory-management"
                 port = 2003
                 hostname = "graphite"
                 enabled = true
                 emitterPoolSize = 1
                 metric {
                   emitter = "com.comcast.money.emitters.GraphiteMetricEmitter"
                 }
             }
         }

• app-name - the namespace that metrics will be reported on
• port - the port of the graphite server
• hostname - the host or ip address of graphite
• enabled - should be set to true
• emitterPoolSize - how many graphite emitters are used to send data. This defaults to 1, but you can increase it if you see memory issues
• metric - leave this as you see it

Kafka Emitter

The Kafka Emitter allows you to send span data to a Kafka cluster. The data is encoded using the Money avro format.

Configuration Example

      {
        name = "kafka-emitter"
        class-name = "com.comcast.money.kafka.KafkaEmitter"
        subscriptions = [Trace]
        configuration = {
          topic = "money"
          compression.codec = "1" // gzip compression enabled by default
          producer.type = "async" // async does not block, but will allow message loss in a network partition
          batch.num.messages = "1" // batching messages together increases throughput
          message.send.max.retries = "3" // retrying on a network partition could introduce duplicates
          request.required.acks = "0" // this indicates that we never wait for an ack.  1 gets an ack once the leader replica receives the data.  -1 waits until all replicas get data
          metadata.broker.list = "localhost:9092"
        }
      }```

## Default Configuration

money {

the name of the application, reported out as app-name in every span

application-name = "unknown"

allows an app to dynamically choose the money config based on environment

override-environment-variable = "MONEY_ENV"

an override for the name of the host

override-hostname-variable = "MONEY_HOSTNAME"

whether money is enabled

enabled = true

the maximum amount of time for a span to remain open before it is forced to close

span-timeout = 60 seconds

the amount of time after a span stops before it is cleaned up and emitted

stopped-span-timeout = 100 ms

should we log exceptions that are caught by money handlers?

log-exceptions = false

enables exposing money data as MDC values for logging purposes

mdc { enabled = true }

enables tracing, this is typically always on

tracer { enabled = true }

all akka configuration that is used by money, you can override anything here

akka { loglevel = INFO log-config-on-start = off loggers = ["akka.event.slf4j.Slf4jLogger"] logging-filter = "akka.event.slf4j.Slf4jLoggingFilter" jvm-exit-on-fatal-error = off }

emitters that are used to emit span data, by default LogEmitter and Span Metrics

emitter { emitters = [ { name = "log-emitter" class-name = "com.comcast.money.emitters.LogEmitter" subscriptions = [Trace] configuration = { emitter = "com.comcast.money.emitters.LogEmitter" } }, { name = "span-metrics-emitter" class-name = "com.comcast.money.metrics.SpanMetricsCollector" subscriptions = [Trace] configuration = {

    }
  }
]

} }```