Kinesis put record error

[rgsriram]

I am keep getting this error.
streams/client.go:97 retrying 5 events on error: failed to put records: InvalidParameter: 1 validation error(s) found. minimum field size of 1, PutRecordsInput.Records

Below is my filebeat.yml file. I couldn't figure out what is the error.

filebeat.prospectors:
   input_type: log
    paths:
    /tmp/test_beat/*.log
       json.keys_under_root: true
           json.add_error_key: true
processors:
- add_cloud_metadata:

output:
  streams:
    enabled: true
    region: XX
    stream_name: XX
    shard_count: XX
    partition_key: XX
    access_key_id: XXX
    secret_access_key: XXX

queue.mem:
  events: 100
  flush.min_events: 5
  flush.timeout: 3s

logging.level: info
logging.to_syslog: false
logging.to_files: true
logging.files:
  path: /tmp/
  name: filebeat.log

`

[mumoshu]

@rgsriram Thanks for reporting!

I think this happens whenever the underlying beat is unable to produce valid events for your logs. So, my best bet is you need to fix your log format somehow.

To see what's causing the issue, you can enable debug logs so awsbeat dumps relevant events and records that helps further investigating the cause.

awsbeats/streams/client.go

Line 90 in 5b58657

logp.Debug("kinesis", "mapped to records: %v", records)

[mumoshu]

You can add an command-line flag -d '*' to the beat to enable debug logging.

[rgsriram]

Thanks for the reply.

…

On Sun, Jan 6, 2019 at 11:46 AM KUOKA Yusuke ***@***.***> wrote: You can add an command-line flag -d '*' to the beat to enable debug logging. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#50 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGvZR7r_tTrSMe4hY8CunVDRrXUA6P0mks5vAZTIgaJpZM4YSK8n> .

-- *Sriram G* *Tech*

[superwhykz]

Shouldn't it be agnostic on message format? Filebeat does not care what it is shipping if it is ascii files.

[superwhykz]

my debug log looks like this

2019-03-05T22:47:45.241Z	DEBUG	[kinesis]	streams/client.go:87	received events: [{{2019-03-05 22:47:40.928463896 +0000 UTC m=+0.034389025 null {"beat":{"hostname":"ip-172-31-37-3","name":"ip-172-31-37-3","version":"6.6.1"},"host":{"name":"ip-172-31-37-3"},"log":{"file":{"path":"/home/ubuntu/beats.log"}},"message":"{'name':'helllo world'}","meta":{"cloud":{"availability_zone":"us-east-2c","instance_id":"i-06b8474a7736caa81","machine_type":"t2.micro","provider":"ec2","region":"us-east-2"}},"offset":0,"source":"/home/ubuntu/beats.log"} { false 0xc00018cdd0 /home/ubuntu/beats.log 24 2019-03-05 22:47:40.927222619 +0000 UTC m=+0.033147766 -1ns log map[] 256333-51713}} 1}]
2019-03-05T22:47:45.241Z	DEBUG	[kinesis]	streams/client.go:114	failed to map event({{2019-03-05 22:47:40.928463896 +0000 UTC m=+0.034389025 null {"beat":{"hostname":"ip-172-31-37-3","name":"ip-172-31-37-3","version":"6.6.1"},"host":{"name":"ip-172-31-37-3"},"log":{"file":{"path":"/home/ubuntu/beats.log"}},"message":"{'name':'helllo world'}","meta":{"cloud":{"availability_zone":"us-east-2c","instance_id":"i-06b8474a7736caa81","machine_type":"t2.micro","provider":"ec2","region":"us-east-2"}},"offset":0,"source":"/home/ubuntu/beats.log"} { false 0xc00018cdd0 /home/ubuntu/beats.log 24 2019-03-05 22:47:40.927222619 +0000 UTC m=+0.033147766 -1ns log map[] 256333-51713}} 1}): failed to get parititon key: failed to get parition key: key not found
2019-03-05T22:47:45.241Z	DEBUG	[kinesis]	streams/client.go:90	sent 0 records: []
2019-03-05T22:47:45.241Z	DEBUG	[kinesis]	streams/client.go:94	mapped to records: []
2019-03-05T22:47:45.241Z	INFO	streams/client.go:101	retrying 1 events on error: failed to put records: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, PutRecordsInput.Records.

[abdalians]

I seem to be getting the same error..
Aug 11 16:48:10 master metricbeat-v7.5.0-go1.11-linux-amd64: 2020-08-11T16:48:10.635Z#011INFO#011streams/client.go:104#011retrying 16 events on error: failed to put records: InvalidParameter: 1 validation error(s) found. Aug 11 16:48:10 master metricbeat-v7.5.0-go1.11-linux-amd64: - minimum field size of 1, PutRecordsInput.Records. Aug 11 16:48:10 master metricbeat-v7.5.0-go1.11-linux-amd64: 2020-08-11T16:48:10.635Z#011INFO#011[publisher]#011pipeline/retry.go:162#011Drop batch

[abdalians]

I solved this with a clue from the metricbeat.yml file in the project repo..

go get github.com/rs/xid
then run make
and add the partition_key_provider: xid to metricbeat.yml in output.streams