Async Lambda Extension (ActiveJob, Suckerpunch, Async NewRelic, Etc)

[metaskills]

If you use :async it will not work on Lambda. Maybe we can use Lambda Extensions to buffer and run things?

https://twitter.com/julian_wood/status/1396882542737903624

[metaskills]

New early return too. https://lumigo.io/blog/lambda-extensions-just-got-even-better/

[metaskills]

https://twitter.com/julian_wood/status/1396935816325255168

[metaskills]

https://twitter.com/dhruvsood/status/1396987415340847106

[nitsujri]

For ActiveJob adapter to use :async (config.active_job.queue_adapter = :async), gives 2 main scenarios:

Non-local (prod, stage)
I wouldn't run this in prod/stage given Lambdakiq. Similar question of sucker_punch vs sidekiq. Given the option, I would always run Sidekiq for the visibility, control, etc. I only run sucker_punch if I am constrained in rather specific ways.

Local
Local dev work does have me use :async, but I'd be hard pressed to tie myself into SQS/Lambdakiq, and thusly Lambda Extensions, for local development.

In relation to Rollbar, log collection, I do agree now that extensions is much more viable though because it replaces sucker_punch as a background threading system. Anytime where I have heavy data && passing to SQS/Lambdakiq isn't viable, background threading rules.

[metaskills]

In relation to Rollbar, log collection, I do agree now that extensions is much more viable though because it replaces sucker_punch as a background threading system. Anytime where I have heavy data && passing to SQS/Lambdakiq isn't viable, background threading rules.

Cool, that's the use case I'm thinking is the strongest too.

[metaskills]

I got pretty far on this today. Lots of reading on Lambda Extensions.

• https://aws.amazon.com/blogs/aws/getting-started-with-using-your-favorite-operational-tools-on-aws-lambda-extensions-are-now-generally-available/
• https://aws.amazon.com/blogs/compute/building-extensions-for-aws-lambda-in-preview/
• https://docs.aws.amazon.com/lambda/latest/dg/runtimes-extensions-api.html
• https://docs.aws.amazon.com/lambda/latest/dg/runtimes-modify.html
• https://docs.aws.amazon.com/lambda/latest/dg/images-create.html
• https://github.com/aws-samples/aws-lambda-extensions/tree/main/ruby-example-wrapper

An "internal" extension is just not going to work. It was easy to create one but in order to keep the same process you will have to use load for the runtime. Seems you can not register for any of the events either. Not sure we need them anyway. The benefit of the internal would have been the same process. So a background unit of work would not need another instance of Rails. The $FOOBAR variable was me debugging the shared process in the handler. Worked great. But a shared process will be just like normal Lambda, frozen and not useful.

#!/var/lang/bin/ruby
$FOOBAR = 'HELLO THERE'
load "/var/runtime/lib/runtime.rb"

I think "external" is the way to go. I got something basic in place using this.

#!/var/lang/bin/ruby

def log(msg) ; puts("['LambdaPunch'] #{msg}") ; end
log "Required"

LAMBDA_PUNCH_BASE_URI = "http://#{ENV['AWS_LAMBDA_RUNTIME_API']}/2020-01-01/extension"
require 'uri'
require 'json'
require 'net/http'

def post(path, options = {})
  uri = URI.parse "#{LAMBDA_PUNCH_BASE_URI}/#{path}"
  http = Net::HTTP.new uri.host, uri.port
  request = Net::HTTP::Post.new uri.request_uri
  request['Content-Type'] = 'application/vnd.aws.lambda.extension+json'
  options[:headers].each do |header, value|
    request[header] = value
  end if options[:headers]
  request.body = options[:body] if options[:body]
  http.request(request)
end

def event_next
  uri = URI.parse "#{LAMBDA_PUNCH_BASE_URI}/event/next"
  http = Net::HTTP.new uri.host, uri.port
  request = Net::HTTP::Get.new uri.request_uri
  request['Content-Type'] = 'application/vnd.aws.lambda.extension+json'
  request['Lambda-Extension-Identifier'] = LAMBDA_PUNCH_EXTENSION_ID
  http.request(request)
end

log "Extension Init.."
response = post 'register', 
                body: %q|{"events":["SHUTDOWN"]}|,
                headers: { 'Lambda-Extension-Name' => File.basename(__FILE__) } 
LAMBDA_PUNCH_EXTENSION_ID = response.each_header.to_h['lambda-extension-identifier']
log "LAMBDA_PUNCH_EXTENSION_ID: #{LAMBDA_PUNCH_EXTENSION_ID}"

event_next

This worked too but in order to call random blocks from the other Rails process in this extension process we need a line of communication. I think the way to go would be to use DRb (https://wiki.c2.com/?DistributedRuby) and (https://ruby-doc.org/stdlib-2.7.1/libdoc/drb/rdoc/DRb.html). So my next step is to test this out and see if it works.

[metaskills]

Fun fact, DRb is used in ActiveSupport's testing parallel stuff. https://github.com/rails/rails/blob/main/activesupport/lib/active_support/testing/parallelization.rb

[metaskills]

As this diagram shows, there is no post-invoke hook. And if we are to use this as an ensure block in the handler to communicate form the app process to the extension, we need some bi-directional IPC (https://www.sitepoint.com/forking-ipc-ruby-part-ii/). But how?

I was reading there is a Log API (https://docs.aws.amazon.com/lambda/latest/dg/runtimes-logs-api.html) as some event system, but that seems very heavy handed and I'm not sure the guarantees are there. Bi-directional IPC with DRb is not really a thing. Sure I could expose state in the app and have the extensions loop till it sees it, but I'm not sure I like that either.

This post on a Rust extension (https://dev.to/aws-builders/building-an-aws-lambda-extension-with-rust-3p81) got me thinking. It talked about inotify as a "sophisticated" approach. I kind of really like that. Ruby has a strong gem to support this (https://github.com/guard/rb-inotify) and what I could do is establish a /tmp/lambdakiq-handled file as an event to signal to the extension it can in turn signal back to the runtime process to process the queue. Some ideas:

• Maybe use the requestId in the file contents (or other data like queue length) as a way to pass data to the extension. This is how the above article used post-invoke hooks.
• When implementing this we would have to add a logical timeout loop in the extension to not poll/wait for the app/runtime process past the timeout. Thankfully the invoke event data gives us deadlineMs.

{
    "eventType": "INVOKE",
    "deadlineMs": 1624399969622,
    "requestId": "59358f44-900e-41f9-ba63-806e585dc418",
    "invokedFunctionArn": "arn:aws:lambda:us-east-1:831702759394:function:lambdapunch-main-Lambda-5iePWQ1aaoaz:live",
    "tracing": {
        "type": "X-Amzn-Trace-Id",
        "value": "Root=1-60d26025-01eef3a30f72afd16dfb2982;Parent=7905e3756d42aff7;Sampled=0"
    }
}

[metaskills]

All done! v1.0.0 released to RubyGems. I've run this thru half a million invokes and everything is working great. Please let me know if you try it and have any feedback.

https://github.com/customink/lambda_punch