Metrics plugin

[pmuens]

Makes it possible to get metrics about the service.
The user can see service metrics with the help of the serverless info command.

Metrics:

• Function runtime
• Invocations
• Throttles
• Errors
• Percentile

Questions:

• Is this a global plugin or AWS related? (Do we need both?)
• Is this service wide of function wide?

What do you guys think @serverless/vip

/cc @flomotlik @eahefnawy

[HyperBrain]

In general I think this is a great idea. Whether it should be global or AWS related depends on the availability of similar metric services in the other providers' infrastructures. If it is not much effort, it can be integrated for AWS only first, and then published via a generic interface later.
How would you return the data? IMHO it should be possible to retrieve it in a human readable format as well as some json structures that can be captured into a file. Also transmitting the edge values for at least the runtime (min, max, sum and average) would be great.
For comparison, see the AWS CLI command for metrics,
With "pricing", do you mean the billed amount of time, or the real credits consumed?

[eahefnawy]

@HyperBrain probably the "billed amount of time" ... It'll probably depend on AWS API, but not sure how accurate that might end up.

[pmuens]

Thanks for the feedback!

I think I'll create a metrics plugin and another one for AWS which will hook into the metrics:metrics lifecycle (like we've done with the other plugins before).

Additionally I'll use the https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/CloudWatch.html API to get the metrics.

We'll start to display the data in a human readable way on the console. So if the user enters serverless metrics --function hello --stage development --region us-east-1 he'll get all the metrics for the corresponding function.

Like with #1654 we're not supporting service wide metrics in the beginning.

[flomotlik]

Currently thinking if this should be prioritised lower and done at some point in the future. @serverless/vip would you guys want to use this plugin during development? Would providing you these metrics help with development at all?

[erikerikson]

@flomotlik prioritization would need to be in a context (relative to what?) but my instinct is that this is on the lower priority side. I can see this being useful but a vital needs use case doesn't occur to me off the top of my head.

[pmuens]

Since we're about to implement this soon it would be great to get some feedback on what features this plugin should have / what would be useful for you @serverless/vip

Thanks in advance! 💯 💃

[erikerikson]

This plugin might be a nice opportunity to create the space in which a "correct means of calculating account region lambda concurrency" might be collected and curated. This is a metric that I know a number of people here would be glad to build some monitoring and automation around.

[RobGruhl]

I'm less interested in generic metrics and more interested in smart alerting - lambda concurrency, high retries, delays, unusual outlier latencies, increased durations, etc. Also linking outlier metrics to specific logs would be nice.

[RobGruhl]

We've been using sum of lambda duration as a poor man's lambda concurrency calculation.

[pmuens]

Thanks for the great input! 👍

Really like the idea of setting up alerts as well. Should the plugin do this automatically or should the user take care of that?