AWS X-Ray Instrumentation

[michaelcohenunsw]

Hi,

I am trying to troubleshoot my background jobs which are timing out most of the time.

I am wondering if anyone has implemented AWS X-Ray with the Lambdakiq hander to aid with troubleshooting a Lambdakiq job?

[metaskills]

Hey @michaelcohenunsw, sorry for the slow reply. Busy week!

I've always found X-Ray hard to implement. I've used it in one Rails application and the overhead of adding the segments, etc felt like work that was solved in other vendors without me having to think about it. So rather than writing my own segments and annotations, I would rather install New Relic and get their automatic instrumentation.

That said, if you think your timeouts are related to an AWS or some other service call and you wanted to use X-Ray, you could opt for the simple SDK/Net::HTTP patching (https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-ruby-patching.html) to see what information it turns up.

Another different option would be instrumenting your code using AWS Embedded Metrics (https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format.html) which sounds fancy but it is basically creating your own data points (just like AWS does) by formatting some simple JSON to STDOUT/CloudWatch. This is how we get all the good ActiveJob events data (https://github.com/customink/lambdakiq#observability-with-cloudwatch) for you. This gem (https://github.com/customink/aws-embedded-metrics-customink) can help using it. Very useful fo apps that do not use New Relic and need some good data like (p95, p99) percentiles for sections of your code. Hope that helps.

[metaskills]

Same, let me know how that goes and what you find out... if anything.

[michaelcohenunsw]

The app is running on ECS and the background jobs on Lambda. The app.rb:

require 'aws-sdk-ssm'

stage = if ENV['RAILS_ENV'] == "development"
  "dev"
elsif ENV['RAILS_ENV'] == "production"
  "prod"
else
  ENV['RAILS_ENV']
end

# Get secrets from AWS SSM because we can not pass them in to the lambda function securely using CDK

s3_options = {
  region: ENV['AWS_REGION']
}

ssm_client = Aws::SSM::Client.new(**s3_options)

secrets = {
  MYSQL_USER: "/#{stage}/REDACTED",
  MYSQL_PASSWORD: "/#{stage}/REDACTED",
  EMAIL_HOST_USER: "/#{stage}/REDACTED",
  EMAIL_HOST_PASSWORD: "/#{stage}/REDACTED",
  SECRET_KEY_BASE: "/#{stage}/REDACTED"
}

# Set environment variables using SSM parameters

secrets.each do |env_name, parameter_name|
  resp = ssm_client.get_parameter({
    name: parameter_name,
    with_decryption: true,
  })

  ENV[env_name.to_s] = resp.parameter.value
end

if !ENV['MYSQL_PORT'].empty?
  ENV['MYSQL_PORT'] = "3306"
end

ENV['RAILS_LOG_TO_STDOUT'] = "1"
ENV['DATABASE_URL'] = "mysql2://#{ENV['MYSQL_USER']}:#{ENV['MYSQL_PASSWORD']}@#{ENV['MYSQL_HOST']}:#{ENV['MYSQL_PORT']}/#{ENV['MYSQL_DATABASE']}?pool=5&timeout=5000"


require_relative 'config/application'
require_relative 'config/environment'
require 'json'
require 'logger'


def handler(event:, context:)
  logger = Logger.new($stdout)
  logger.info(event.to_json)
  Lambdakiq.handler(event)
end

[metaskills]

I wonder if things are timing out because of the SSM gets?

[michaelcohenunsw]

Thanks for the suggestion. I will add some more logging around the SSM gets and see what happens...

[metaskills]

Cool, one common technique is to bundle the configs with the deploy package and remove coupling Lambda init to other services all together. See here. Lamby has a rake task that can help. This is why our starter uses Dotenv too.

https://github.com/customink/lamby-cookiecutter-ci/blob/master/%7B%7Bcookiecutter.project_name%7D%7D/bin/_build#L12-L16

I like simple. But another complex option would be to do something like this too. https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/cache-secrets-using-aws-lambda-extensions.html