[CONTP-1014] feat: Add UST docker labels to all container definitions

[Mathew-Estafanous]

What does this PR do?

Adds UST docker labels to all app container definitions excluding agent containers.

configuration to docker label mapping:

• dd_env -> com.datadoghq.tags.env
• dd_service -> com.datadoghq.tags.service
• dd_version -> com.datadoghq.tags.version

Exposes configurable docker labels on agent containers through dd_docker_labels.

Motivation

ecs.fargate.cpu.limit and similar metrics don't have service/env/version tags despite setting UST env vars on all container environments. That's because UST tags cannot be created based on environment variables when deploying on ECS Fargate.

Adding UST docker labels fixes this such that metrics and logs are properly tagged with service/env/version.

Describe how you validated your changes

• Added ust docker label tests ust_docker_labels_test.go.

Manual Testing

1. Terraform apply with dd_service/dd_version/dd_env setup.

module "datadog_ecs_fargate_task" {
  source = "../../modules/ecs_fargate"

  # Configure Datadog
  dd_api_key                       = var.dd_api_key
  dd_site                          = var.dd_site
  dd_tags                          = "team:cont-p, owner:container-monitoring"
  dd_essential                     = true
  dd_is_datadog_dependency_enabled = true

  dd_service = "test-service"
  dd_env     = "test"
  dd_version = "1.2.3"

  dd_environment = [
    {
      name  = "DD_CUSTOM_FEATURE",
      value = "true",
    },
  ]

  dd_dogstatsd = {
    enabled                  = true
    dogstatsd_cardinality    = "high",
    origin_detection_enabled = true,
  }

  dd_apm = {
    enabled   = true,
    profiling = true,
  }

  dd_log_collection = {
    enabled = true,
  }

  dd_cws = {
    enabled = true,
  }

  # Configure Task Definition
  family = var.task_family_name
  container_definitions = jsonencode([
    {
      name      = "dummy-dogstatsd-app",
      image     = "ghcr.io/datadog/apps-dogstatsd:main",
      essential = false,
    },
    {
      name      = "dummy-apm-app",
      image     = "ghcr.io/datadog/apps-tracegen:main",
      essential = true,
    },
    {
      name      = "dummy-cws-app",
      image     = "public.ecr.aws/ubuntu/ubuntu:22.04_stable",
      essential = false,
      entryPoint = [
        "/usr/bin/bash",
        "-c",
        "cp /usr/bin/bash /tmp/malware; chmod u+s /tmp/malware; apt update;apt install -y curl wget; /tmp/malware -c 'while true; do wget https://google.com; sleep 60; done'"
      ],
    }
  ])
  volumes = [
    {
      name = "app-volume"
    }
  ]
  runtime_platform = {
    cpu_architecture        = "ARM64"
    operating_system_family = "LINUX"
  }
  requires_compatibilities = ["FARGATE"]
}

2. Deploy task definition as service.
3. Confirm ecs.fargate.cpu.limit metrics are tagged with UST tags.

Additional Notes

[gabedos]

Other application container configured options like environment variables, mount points, and container dependencies give more precedence to the values defined by the Datadog Terraform module.

I think we should be fine to maintain that level of precedence here and assume that if a user defined the UST values for env, service, and version then those values would take precedence as well.

terraform-aws-ecs-datadog/modules/ecs_fargate/variables.tf

Lines 117 to 133 in 1419f73

	variable "dd_service" {
	description = "The task service name. Used for tagging (UST)"
	type = string
	default = null
	}
	variable "dd_env" {
	description = "The task environment name. Used for tagging (UST)"
	type = string
	default = null
	}
	variable "dd_version" {
	description = "The task version name. Used for tagging (UST)"
	type = string
	default = null
	}

We can leave a bugfix note in the changelog for the next release to notify users that the UST tagging has been fixed.

[Mathew-Estafanous]

Understandable. I guess if they wanted unique UST values, then they shouldn't set the global env,service,and version. They should then manually set it per container definition.

I initially wanted to prioritize the user's docker labels over ours because the customer's problem description implied the need for different UST values.

[gabedos]

I'm unsure if we should be setting the UST tags on the Agent container the same as the application containers. We should get some other opinions about this and compare what our approach is for EKS Fargate with the admission controller on the DCA.

[Mathew-Estafanous]

We already set UST env vars, albeit they don't work for the agent.

The DCA agent sidecar mutator doesn't inject the agent with any UST tags, so I think It makes sense to remove these UST tags from the terraform agent as well.

[gabedos]

Correct, the UST env vars on the Agent (and also fluentbit and cws-init) are actually doing nothing. We should be fine clean up and remove them.