CloudWatch sink group_name can't contain templated variables during validate

[archoversight]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Vector Version

vector 0.19.0 (x86_64-unknown-linux-gnu da60b55 2021-12-28)

Vector Configuration File

[sinks.cloudwatch]
type = "aws_cloudwatch_logs"
inputs = ["*__out"]
create_missing_group = true
create_missing_stream = true
group_name = "{{ file }}"
compression = "none"
region = "us-east-1"
stream_name = "{{ host }}"
encoding.codec = "json"

Debug Output

# VECTOR_CONFIG_DIR=/etc/vector/ns2/,/etc/vector/sinks/ VECTOR_CONFIG=/etc/vector/vector.toml RUST_BACKTRACE=full vector -v validate
√ Loaded ["/etc/vector/vector.toml", "/etc/vector/ns2", "/etc/vector/sinks"]
√ Component configuration
2022-01-11T04:47:16.764139Z ERROR vector::topology::builder: msg="Healthcheck: Failed Reason." error=DescribeLogGroups failed: 1 validation error detected: Value '{{ file }}' at 'logGroupNamePrefix' failed to satisfy constraint: Member must satisfy regular expression pattern: [\.\-_/#A-Za-z0-9]+ component_kind="sink" component_type="aws_cloudwatch_logs" component_id=cloudwatch component_name=cloudwatch
x Health check for "cloudwatch" failed

Expected Behavior

vector validate should continue without issues

Actual Behavior

vector validate fails

Example Data

Not Applicable

Additional Context

Not Applicable

References

None

[spencergilbert]

Similar to #10465, feels like we need to add a default in these cases or setup a healthcheck that doesn't require the group_name/topic to be present for the check.

[BryanStenson-okta]

I think it's worth mentioning the repro config file @archoversight uses contains the example, suggested templating details for group_name and stream_name:

https://vector.dev/docs/reference/configuration/sinks/aws_cloudwatch_logs/#group_name (see examples)
https://vector.dev/docs/reference/configuration/sinks/aws_cloudwatch_logs/#stream_name (see examples)

[BryanStenson-okta]

fwiw, despite the validation error, i'm still seeing logs shipped to cloudwatch. (image: timberio/vector:0.19.0-distroless-libc)

[spencergilbert]

fwiw, despite the validation error, i'm still seeing logs shipped to cloudwatch. (image: timberio/vector:0.19.0-distroless-libc)

👍 unless you've required healthcecks to pass the failure won't block running (--require-healthy)

[archoversight]

By default the systemd won't let vector continue starting if the prestart command fails, so I had to explicitly remove the pre-start command.

[mmerickel]

Bumping this as I ran into it as well on vector 0.26.0. It definitely is a bad default as group_name is documented to support templates. The healthcheck needs to account for that and do something else.

2022-12-23T00:53:51.225148Z  INFO vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2022-12-23T00:53:51.225310Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,lapin=info,kube=info"
2022-12-23T00:53:51.225387Z  INFO vector::app: Loading configs. paths=["/etc/vector"]
2022-12-23T00:53:51.227185Z  INFO vector::sources::kubernetes_logs: Obtained Kubernetes Node name to collect logs for (self). self_node_name="ip-10-34-6-71.us-east-2.compute.internal"
2022-12-23T00:53:51.239782Z  INFO vector::sources::kubernetes_logs: Excluding matching files. exclude_paths=["**/*.gz", "**/*.tmp"]
2022-12-23T00:53:51.307101Z  INFO vector::topology::running: Running healthchecks.
2022-12-23T00:53:51.311137Z  INFO vector: Vector has started. debug="false" version="0.26.0" arch="x86_64" revision="c6b5bc2 2022-12-05"
2022-12-23T00:53:51.311388Z  INFO vector::internal_events::api: API server running. address=0.0.0.0:8686 playground=off
2022-12-23T00:53:51.319409Z  INFO source{component_kind="source" component_id=kubernetes_logs component_type=kubernetes_logs component_name=kubernetes_logs}:file_server: file_source::checkpointer: Loaded checkpoint data.
2022-12-23T00:53:51.377168Z ERROR vector::topology::builder: msg="Healthcheck: Failed Reason." error=DescribeLogGroups failed: InvalidParameterException: 1 validation error detected: Value '/aws/eks/main/vector/test/{{ kubernetes.pod_namespace
 }}' at 'logGroupNamePrefix' failed to satisfy constraint: Member must satisfy regular expression pattern: [\.\-_/#A-Za-z0-9]+ component_kind="sink" component_type="aws_cloudwatch_logs" component_id=cloudwatch component_name=cloudwatch

[birnbera]

Just wanted to bump this as the issue is still happening with vector 0.44.0 and can confirm that having the health check fail as part of the service ExecStartPre/ExecReload means that you can't use systemd to run vector with the aws_cloudwatch_logs sink if it uses templates in the group_name.

[iandelahorne]

Just hit this on 0.54.0 as well. Excerpt from cloudtrail:

{

"eventSource": "logs.amazonaws.com",
"eventName": "DescribeLogGroups",
"errorCode": "InvalidParameterException",
"errorMessage": "1 validation error detected: Value '{{ cloudwatch_log_group_name }}' at 'logGroupNamePrefix' failed to satisfy constraint: Member must satisfy regular expression pattern: [\\.\\-_/#A-Za-z0-9]+",
"requestParameters": {
  "logGroupNamePrefix": "{{ cloudwatch_log_group_name }}",
  "limit": 1
  }
}

Config:

sinks:
  cloudwatch:
    type: aws_cloudwatch_logs
    inputs:
      - container_logs
    region: "${AWS_REGION:-us-east-1}"
    group_name: "{{ cloudwatch_log_group_name }}"
    stream_name: "{{ cloudwatch_log_stream_prefix }}/{{ cloudwatch_host }}"
    create_missing_group: false
    encoding:
      codec: json