Kinesis output plugin does not work in govcloud

[skroll]

Bug report

Relevant telegraf.conf:

[[outputs.kinesis]]
  region = "us-gov-west-1"
  streamname = "my-metrics-stream"
  partitionkey = "public"
  use_random_partitionkey = false
  data_format = "influx"
  debug = false

System info:

Telegraf v1.3.0
Amazon Linux

Steps to reproduce:

1. Set an kinesis output to write telegraf metrics.
2. Start telegraf.

Expected behavior:

Telegraf to start and write metrics to the stream.

Actual behavior:

Telegraf fails to start, prints out that the 'Endpoint' configuration is required since it cannot resolve it automatically.

Additional info:

2017-06-12T15:27:19Z E! kinesis: Error in ListSteams API call : MissingEndpoint: 'Endpoint' configuration is required for this service
2017-06-12T15:27:19Z E! kinesis : You have configured a StreamName my-metrics-stream which does not exist. exiting.

[danielnelson]

Do you know if it works outside of the govcloud?

[skroll]

Yes it works fine. It appears that the go AWS SDK can't resolve the endpoint correctly when it's in govcloud, since they do not obey the same naming convention for the URLs.

[skroll]

All that really needs to be done is exposing a configuration value for the endpoint to be set in the AWS service configuration. I did something like:

var svc *kinesis.Kinesis
if k.Endpoint != "" {
	svc = kinesis.New(configProvider, aws.NewConfig().WithEndpoint(k.Endpoint))
} else {
	svc = kinesis.New(configProvider)
}

This is definitely something I can contribute, I'm just not sure about the go style conventions used by telegraf.

[danielnelson]

That looks good.

[skroll]

Ok, actually, it's due to the version of the aws-sdk-go dependency.

The commit listed in Godeps was before Kinesis was added to govcloud, so it doesn't have a resolver for kinesis when in us-gov-west-1.

Newer versions of the SDK resolve this endpoint automatically.

[danielnelson]

Thanks, I'll update the dependency.

[danielnelson]

I updated the dependency, give it a spin if you can.

[skroll]

I'll fire it up tomorrow and report back.

[skroll]

Confirming that this works now, with the exact same config file I was using before.