Create metric for worker lambda processing rate

[DavidLawes]

What does this change?

We currently use metrics to see the performance of the whole system. For worker lambdas we have a metric for the processing time, which includes the length of time a message sits on a queue. This is great for determining the impact of changes to the overall performance.

In future PRs we are going to test using batch api calls. This PR adds a new metric to enable us to evaluate improvements to the lambda function.

The functionProcessingRate is a metric for how quickly we process tokens. The rate is calculated by:

number of tokens processed by a particular function invocation / function processing time in seconds

The result is that we get a metric representing the number of tokens being processed per second.

To capture this metric there were a number of changes required:

• Refactor the sender request handler to abstract logging code into Logger class
• Capture the start time of the function (we actively didn't want to use the sentTime, i.e. when the message landed on the queue, instead capturing when the function starts)
• Refactor when we determine the function has completed. Through local testing we could see we create ("compile") our stream before executing it. We changed where we capture the end time to ensure this happened when a chunk of tokens had actually completed sending.
• Calculate the function processing rate using the function start time, end time and number of tokens
• Extend cloudwatch embedded metrics to include new metric for function processing rate
• Update local run to facilitate testing of embedded metrics

How to test

Locally we can test to ensure the lambda compiles and runs successfully. We can see in the log messages that the relevant data was being created.

To test the new embedded metric the lambda logs needed to be parsed by AWS. I deployed to CODE and checked that we could see the lambda metric being recorded via the AWS console.

[DavidLawes]

I can currently see the metrics appearing in aws cloudwatch, but the metric values are resolving to 0, always:

This is different to the raw json value that's been parsed, which we can see in kibana:

Will continue to investigate, but for reference the json log generated parses successfully against the schema aws provide for embedded metrics.

EDIT: I tried to be helpful by defining the decimal places of the rate, but this converted the type to a string, aws only accepts numbers. Now the metric is parsed as expected and is viewable in the console (non-zero):

[waisingyiu]

Thank @DavidLawes for the great work!