Kinesis Agent has high CPU for cases with many files

[frankfarrell]

Our use case is to SFTP a file to our server every minute, and Kinesis agent is configured to match these files. The files are not modified after they are written and they have a footer line, eg "File Administratively Closed..."

Using jstack we identified the thread using high CPU as

"FileTailer[<filePattern>]" #13 prio=5 os_prio=0 tid=0x00007f2bf0574000 nid=0xdadb runnable [0x00007f2bcc490000]
   java.lang.Thread.State: RUNNABLE
    at java.lang.Object.hashCode(Native Method)
    at java.util.HashMap.hash(HashMap.java:338)
    at java.util.HashMap.put(HashMap.java:611)
    at com.amazon.kinesis.streaming.agent.tailing.TrackedFileRotationAnalyzer.syncCounterpartsByFileId(Unknown Source)
    at com.amazon.kinesis.streaming.agent.tailing.TrackedFileRotationAnalyzer.<init>(Unknown Source)
    at com.amazon.kinesis.streaming.agent.tailing.SourceFileTracker.updateCurrentFile(Unknown Source)
    at com.amazon.kinesis.streaming.agent.tailing.SourceFileTracker.refresh(Unknown Source)
    at com.amazon.kinesis.streaming.agent.tailing.FileTailer.updateRecordParser(Unknown Source)
    - locked <0x00000000eac78af0> (a com.amazon.kinesis.streaming.agent.tailing.FileTailer)
    at com.amazon.kinesis.streaming.agent.tailing.FileTailer.processRecords(Unknown Source)
    - locked <0x00000000eac78af0> (a com.amazon.kinesis.streaming.agent.tailing.FileTailer)
    at com.amazon.kinesis.streaming.agent.tailing.FileTailer.runOnce(Unknown Source)
    at com.amazon.kinesis.streaming.agent.tailing.FileTailer.run(Unknown Source)
    at com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:60)
    at com.google.common.util.concurrent.Callables$3.run(Callables.java:95)
    at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
    - None 

Possible Solution:
A solution would be to have a fileFooterPattern that stops tailing the file when it is matched. This has the advantage of not modifying the file (eg, as would adding .CLOSED to the file).

[johnou]

@frankfarrell try increasing the value of minTimeBetweenFilePollsMillis to 1000 (default is 100).

[johnou]

@chaochenq please consider swapping the thread per file busy loop implementation with WatchService (supports registering multiple paths etc.).

http://docs.oracle.com/javase/tutorial/essential/io/examples/WatchDir.java