Datadog agent integration

[timotheeg]

Context

We are implementing DataDog as a central monitoring system. The FormSg AWS account has been integrated to DataDog already, but more needs to be done:

• Run DataDog Agent on our servers
• Install various DataDog connectors (e.g. for express and winston)
• Report custom data as needed

This PR only install the DataDog agent on the FormSG servers.

Approach

Run the DataDog agent on the FormSG server. It will report host system metrics.

The EB environment needs 2 new environment variables: DD_API_KEY (see here) and DD_ENV (staging|production).

Tested on staging, sample data here:
https://app.datadoghq.com/dashboard/b3u-egb-apu/formsg?tpl_var_env=staging&from_ts=1632260114510&to_ts=1632274514510&live=true

Note: The DD agent is configured to accept non-local traffic to allow host-Docker communication. Care should be taken to never expose the DogStatsD port 8125/8126(?) to other hosts, or the internet.

Deploy Notes

• Early (e.g. 730am or 8am) add the 2 environment variables below (cause EB redeploy, but new env vars are not used yet)
• Deploy new as normal

New environment variables:

• DD_API_KEY : FormSG specific API key for DataDog
• DD_ENV: staging or production

New dependencies:

• hot-shots : client for StatsD/Datadog Agent
• connect-datadog: integrations for Connect/Express

[mantariksh]

Early (e.g. 730am or 8am) add the 2 environment variables below (cause EB redeploy, but new env vars are not used yet)

@timotheeg can I check why we need to do this? adding environment variables is safe, we do it regularly (e.g. to update site banners)

[timotheeg]

Early (e.g. 730am or 8am) add the 2 environment variables below (cause EB redeploy, but new env vars are not used yet)

@timotheeg can I check why we need to do this? adding environment variables is safe, we do it regularly (e.g. to update site banners)

As we discussed earlier, this was me being paranoid. Changing env vars causes an EB deployment, which means old instances are put in drain mode. That is safe, provided we do not have long running requests which exceed the drain timeout (I haven't checked what's the drain timeout for our setup btw). Such long requests can break, so doing an early change when there's low traffic is just reducing risks. Since the majority of our requests are short, that note may indeed be overkill. But ideally, we have predictable response times for all requests (or built-in retries mechanisms for known long requests), guaranteeing that changing env vars is indeed completely safe at any time.

Btw, the same issue of possibly breaking long requests applies for all deploys.

[liangyuanruo]

I would also note that enabling non-local traffic to allow host-Docker communication also means that we should never expose the DogStatsD port 8125/8126(?) to the internet.

[timotheeg]

I would also note that enabling non-local traffic to allow host-Docker communication also means that we should never expose the DogStatsD port 8125/8126(?) to the internet.

That is correct. I don't think think we would ever open these ports (?). In fact they should not even be opened to any other hosts in the network.

[tshuli]

lgtm except one nit

[mantariksh]

lgtm, one question