[Filebeat] Adding wildcard support for log_group_name in cloudwatch input

[srolskyi]

Find awscloudwatch log groups by discovery(pattern)

For example I have ~400 log_groups and I should to parse only ~70
My log_group contain some similar names like:

/development/application/api-south
/development/application/api-west
/development/application/web-south
.....

My filebeat.yml configuration:

- type: awscloudwatch
  access_key_id: 0000000000000000000
  secret_access_key: 0000000000000000000
  log_group_name: /development/application/
  region_name: us-east-1
  scan_frequency: 10s
  start_position: beginning

Got an error:

2020-10-05T14:36:28.816Z	INFO	[awscloudwatch]	awscloudwatch/input.go:125	awscloudwatch input worker for log group: '/development/application/' has started
2020-10-05T14:36:28.816Z	INFO	[crawler]	beater/crawler.go:108	Loading and starting Inputs completed. Enabled inputs: 1
2020-10-05T14:36:28.816Z	INFO	cfgfile/reload.go:164	Config reloader started
2020-10-05T14:36:28.816Z	INFO	cfgfile/reload.go:224	Loading of config files completed.
2020-10-05T14:36:29.337Z	ERROR	[awscloudwatch]	awscloudwatch/input.go:189	failed FilterLogEventsRequestResourceNotFoundException: The specified log group does not exist.
	status code: 400, request id: 6fb32149-c2d1-49e3-9f41-f5a30810fae3

If I set: log_group_name: /development/application/* have similar issue:

getLogEventsFromCloudWatch failed: InvalidParameterException: 1 validation error detected: Value ':/development/application/*' at 'logGroupName' failed to satisfy constraint: Member must satisfy regular expression pattern: [\.\-_/#A-Za-z0-9]+

So for my configuration, I must set up 70 input configurations (for each log_group)? Or I lost something and I may use some special pattern?

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[kaiyan-sheng]

@srolskyi Thank you for reaching out! Yes we currently don't have the support for using wildcard in log_group_name config yet. I will use this issue to track the work for adding * support for log_group_name.

[EmViKay]

@kaiyan-sheng we have the same issue.
Do you have some deadline for this task?

[EmViKay]

any updates? :)

[sunilmchaudhari]

Hi,
Any ETA for this?
when can we have this feature?

[chyeyun]

@kaiyan-sheng
Hello, This is Hyeyun. I opened the issue case #29006 about that 'log_group_name_prefix' doesn't work.
Could you check that issue, please? Thank you.