Centralized logging with Kinesis Firehose using Lambda Extensions

[hariohmprasath]

Issue #, if available:
Lambda Extension to send logs directly to Kinesis firehose #51

Description of changes:
The extension explains the overall approach to streamline and and centralize log collection using Kinesis firehose. The extension runs a local HTTP listener and subscribes to a stream of function and platform logs using the Logs API. It buffers them and sends them to Kinesis firehose periodically. The demo gets deployed using AWS SAM.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[julianwood]

It may also be worth thinking about sending events per invoke. You can use the Platform runtimeDone messages to in effect wait until all the logs have arrived from Lambda before calling /next. This doesn't slow down the function response and makes for cleaner logs when the execution environment is frozen between invocations which may mean only receiving the logs during the next invoke.

[hariohmprasath]

It may also be worth thinking about sending events per invoke. You can use the Platform runtimeDone messages to in effect wait until all the logs have arrived from Lambda before calling /next. This doesn't slow down the function response and makes for cleaner logs when the execution environment is frozen between invocations which may mean only receiving the logs during the next invoke.

Yep,
The code flushes logs only incase of Shutdown and runtimeDone events. Here is a snippet of it.

flushLogQueue := func(force bool) { for !(logQueue.Empty() && (force || strings.Contains(logsStr, string(logsapi.RuntimeDone)))) { // Flush logs } }

flushLogQueue(true/false)

[julianwood]

Looking good, just some working suggestions in the README for clarity and correct use of service names.

[hariohmprasath]

Looking good, just some working suggestions in the README for clarity and correct use of service names.

Thanks @julianwood, all the changes are incorporated.

[julianwood]

Thanks for contributing.