Fix Lambda Extension logs shutdown

[nhinsch]

What does this PR do?

There is a race condition that causes panics and prevents us from receiving logs when shutting down the Lambda Extension. When the extension receives a shutdown event from the API, it stops the logs agent. However, the extension may continue to receive log messages from the logs API after receiving the shutdown event. Currently, when these messages are received, we try to send them to the logs agent. Since the logs agent has been stopped, this results in a panic.

To fix this issue:

• Consolidate the shutdown logic in a new Daemon.Stop() function.
• Add a 1-second delay after we receive the shutdown signal before shutting down. This should allow for final log messages to be delivered from the logs API.
• After the 1-second delay, shut down the HTTP server first before shutting down the agents. This will prevent us from sending messages to the agents after they have already shut down.
• Do some refactoring so that we also call Daemon.Stop() after receiving a SIGINT signal. This shouldn't really happen in practice because during a shutdown, we receive a shutdown message over the Lambda API, and then SIGKILL two seconds later.

Motivation

The current behavior causes panics and lost log messages.

Describe how to test your changes

I have been manually testing these changes with functions running periodically in the sandbox account that frequently encounter the race condition. I used debug log messages to confirm the behavior is as expected.

[nhinsch]

@DarcyRaynerDD I have refactored this and incorporated your suggestions. I updated the initial PR comment to reflect the new approach. Let me know your thoughts.

[remeh]

Just a comment on the usage of the lock in the shutdown which I think needs one more commit, but except from that, solid!

[remeh]

💯