Temporary File Cleanup Can Cause Kinesis Producer to Fail

[pfifer]

The Kinesis Producer, by default, stores native files in a directory in /tmp. This includes the executable, and certificates used to access Kinesis itself. Temporary cleanup cron jobs can remove these files, and while the Kinesis Producer will still run it will be unable to make TLS connections due to a missing certificates.

Failures of the KPL will include an error similar to below

[AWS Log: ERROR](CurlHttpClient)Curl returned error code 60

If you're affected by this the current best fix is to restart the parent Java process. This will restore the missing files, and return the KPL to normal.

There are a couple of temporary workarounds:

• Disable your temporary file cleanup job
• Keep the certificate file alive with a recently modified date
  Adding this as a cron job should keep the certificate file from being cleaned up

touch /tmp/amazon-kinesis-producer-native-binaries/b204d74a.0

• Restart the Java process on a regular schedule
  After each restart the Java process should replace the files.

[pfifer]

Released new version 0.12.3 that fixes this.

[felipe-vvoosh]

I'm having this issue even using 0.12.5...

[umatrangolo]

Still happening for me with 0.12.7.

2018-05-15 03:17:10,316 [kpl-daemon-0003] ERROR com.amazonaws.services.kinesis.producer.LogInputStreamReader - [2018-05-15 03:17:10.316354] [0x00002ce9][0x00007faf8548370
0] [error] [AWS Log: ERROR](CurlHttpClient)Curl returned error code 60

Bouncing the JVM will fix but is not ideal having downtime for this.

[buremba]

I'm using 0.12.6 and recently run into this problem. The CPU load was around 15% the errors are similar to what @umatrangolo had. I had to restart the JVM to fix the problem but we had downtime because if this bug.