Variables substitution.

[Infra-dev]

What I'm doing wrong?

i wan't use dynamic names for log_group and stream.
Forwarder config look like that

{
  "files": [
    { "paths": [
        "/var/log/nginx/error.log"
      ],
      "fields": {
        "type": "NginxErrorLogs",
        "instance": "i-xxxxxxxxx"
      }
    }
  ]
}

Logstash config looks:

output {
  cloudwatchlogs {
    "log_group_name" => "NginxErrorLogs"
    "log_stream_name" => "%{instance}"
  }
}

And in CWL i see stream name %{instance} but expected to see "i-xxxxxxxxx"

[wanghq]

Hello,

This is not supported. An instance of cloudwatchlogs output only maps to one log stream and the log stream name is set when the plugin is registered so it's not possible to change the log stream name based on the event fields with the current implementation.

In the CloudWatch Logs python agent, {instance_id} is automatically substituted to the actual instance id and we may add that.

Could you move the instance field from input to output? As showed in the example:

output {
  if "app1" in [tags] {
    cloudwatchlogs {
      "log_group_name" => "app1"
      "log_stream_name" => "i-xxxxxxxxx"
    }
  }
  if "app2" in [tags] {
    cloudwatchlogs {
      "log_group_name" => "app2"
      "log_stream_name" => "i-yyyyyyyyy"
    }
  }

[Infra-dev]

Hello, thanks for response.
Yes i can use if statement for sure. But my main problem is paste field instance_id to log_steam_name.

[vaijab]

I was just actually trying various ways to reference a field value in cloudwatchlogs configuration block with no luck.

It would be great if the plugin supported that.

[wanghq]

@vaijab, can you share your use case? That would help us decide whether we should support that. Thanks.

[vaijab]

@wanghq sure.

I have a service, which has a concept of namespaces. I would like to be able to push events to a particular namespace depending on which namespace an event belongs to.

Example event looks like below:

{
    "log": "10.10.91.0 - - [05/Oct/2015:11:17:51 +0000] \"GET /dataset HTTP/1.1\" 200 2 \"-\" \"axios/0.5.4\" 5\n",
    "stream": "stdout",
    "time": "2015-10-05T11:17:51.224167526Z",
    "replication_controller": "data-foo-api",
    "pod": "data-foo-api-p82sy",
    "namespace": "catalogue",
    "container_name": "data-foo-api",
    "container_id": "df1874255f0c85d18747b5edfc8dc372dbebf725b9ccbfb37549f5f81bba8326"
}

I would like to be able to configure logstash like this:

output {
    cloudwatchlogs {
      region => "us-east-1"
      log_group_name => "[namespace]"
      log_stream_name => "[container_name]"
    }
}

For reference:

• logstash kubernetes filter: https://github.com/vaijab/logstash-filter-kubernetes
• logstash configuration (docker container): https://github.com/UKHomeOffice/docker-logstash-kubernetes

[teebu]

i was also trying to do a variable passing. any update on this?

[jhmartin]

It'd be useful to make the logs stream configurable, such as when posting container-based logs -- you'd use one log stream per container, not per host.

[jlestel]

Hello,

I try to use {instance_id} in my output configuration block :
output {
cloudwatchlogs {
"log_group_name" => "access_log"
"log_stream_name" => "{instance_id}"
"region" => "eu-west-1"
}
}

But "{instance_id}" is not replaced and see this in CloudWatch. What is the exact syntax ?

[w3iBStime]

I'd also like to see at least %{instance_id} supported--we have a number of identical queue consumers running in parallel/fanout. Each process should map to it's own sequential log stream.

For now, I'll try working around this issue using the wrapping if block pattern--it'll just be a bit repetitive.

[w3iBStime]

It'd also be nice to have support for %{host} in the stream name but I can programmatically generate a static entry for that in the Logstash config.