Auto-Tag AWS Log Groups to make cost-management easier

[rtyley]

Cloudwatch Logs can be expensive (even in terms of Collect/Data0Ingestion @ $0.57 per GB, if you generate enough GBs...) but without AWS tagging it's difficult for the various teams using the AWS Ophan account to determine which team is responsible for the costs in AWS Cost Explorer:

Tagging AWS Log Groups is fiddly (it can't even be done through the AWS Console, only through AWS CLI & API), but could the cloudwatch-logs-management lambdas automatically perform the tagging?

Here are two logged ELK events, generated by an Ophan and a Data Lake lambda, which correctly have their stack context set - if this information is present in the log event, can't the cloudwatch-logs-management lambda use it to tag the AWS Log Group correctly?

{
  "_index": "logstash-ophan-2019.11.19",
  "_type": "_doc",
  "_id": "NoqLg24B2c_gLWAcdvsB",
  "_score": 1,
  "_source": {
    "message": "2019-11-19 12:02:58 [main]  INFO  WorkerLambda$:113 - Monitoring ran successfully for friction_screen on ios. No problems were detected.\nActual impressions: 72829 | Expected impressions: 40000 | Percentage: 82% above the threshold.",
    "@timestamp": "2019-11-19T12:02:58.208Z",
    "stack": "ophan-data-lake",
    "stage": "PROD",
    "cloudwatchId": "35105052079467119013119283147712778190358612601176326163",
    "cloudwatchLogGroup": "/aws/lambda/data-lake-alerts-worker-PROD",
    "app": "data-lake-alerts",
    "@version": "1",
    "source_stream": "ophan"
  },
  "fields": {
    "@timestamp": [
      "2019-11-19T12:02:58.208Z"
    ]
  }
}

{
  "_index": "logstash-ophan-2019.11.19",
  "_type": "_doc",
  "_id": "rpogg24BXqtOzgEfC5ks",
  "_score": 1,
  "_source": {
    "message": "REPORT RequestId: c20ab503-8297-4e5d-98b2-07b9dad07464\tDuration: 30513.83 ms\tBilled Duration: 30600 ms\tMemory Size: 512 MB\tMax Memory Used: 83 MB\tInit Duration: 314.16 ms",
    "lambdaStats": {
      "memorySizeMB": 512,
      "maxMemoryUsedMB": 83,
      "durationms": 30513.83,
      "initDurationms": 314.16,
      "billedDurationms": 30600
    },
    "lambdaEvent": "REPORT",
    "cloudwatchId": "35104894887089400870389405737696487847887205821691854849",
    "stack": "ophan",
    "cloudwatchLogGroup": "/aws/lambda/enr-remove-node-PROD",
    "app": "ophan-es-node-rotation",
    "source_stream": "ophan",
    "@timestamp": "2019-11-19T10:05:29.458Z",
    "stage": "PROD",
    "lambdaRequestId": "c20ab503-8297-4e5d-98b2-07b9dad0746",
    "@version": "1"
  },
  "fields": {
    "@timestamp": [
      "2019-11-19T10:05:29.458Z"
    ]
  }
}

[andreas-ekman-wcar]

aws-cloudformation/cloudformation-coverage-roadmap#77

[akash1810]

LogGroups have recently gained tagging support 🎉