tail for Kinesis Stream
Go
Latest commit 706ab14 Apr 20, 2016 @suzuken Merge pull request #4 from suzuken/time_based_iterator
support time based shard iterator
Permalink
Failed to load latest commit information.
README.md
main.go

README.md

kinesis-tail

Continuously fetch records from Kinesis Stream and print it. Useful for debugging Kinesis.

How to use

$ go get github.com/suzuken/kinesis-tail
$ kinesis-tail -stream your-kinesis-stream-name -region ap-northeast-1 -max-item-size=512
Your Stream information: {
  StreamDescription: {
    HasMoreShards: false,
    RetentionPeriodHours: 24,
    Shards: [{
        HashKeyRange: {
          EndingHashKey: "340282366920938463463374607431768211455",
          StartingHashKey: "0"
        },
        SequenceNumberRange: {
          StartingSequenceNumber: "49556705040321186153938184833717233207750712554199973890"
        },
        ShardId: "shardId-000000000000"
      }],
    StreamARN: "arn:aws:kinesis:ap-northeast-1:xxxxxxxxxxxx:stream/your-kiness-stream-name",
    StreamName: "your-kiness-stream-name",
    StreamStatus: "ACTIVE"
  }
}
{
  ShardIterator: "AAAAAAAAAAF6jxGy30eyuTD4GzFrqAwEUFdWMIiWtFqpzGIShgXfo40VHDGHzNAdN0uvK5j6llWkEJQtdKZC3kQTIOQ/Y7Zj/lyBrV4GTkqz/hT+m/8TCZHU61fbEqU7Wd9V+Wa9szmeBQfdN5BAZDybhSUctjJRT+mgfbxVPXOjLkIrBNpAvzPFLRRaNSJj/ZwxIcG3k3nfwY7yUrcNb42OR6Cu7o1P"
}
ApproximateArrivalTimestamp: 2015-12-01 02:14:23 +0000 UTC
Data: {"id":"5310e2ef-25e3-4942-87df-2aed28320b99","hoge":"fuga"}
SequenceNumber: 49556846757453585699518614739447433851727187225060835330
ApproximateArrivalTimestamp: 2015-12-01 02:24:41 +0000 UTC
Data: {"id":"75ba2e4a-a68d-4d50-89cf-8b954a115a08","hoge":"fuga"}
SequenceNumber: 49556846757453585699518614739460732035742990683338702850
ApproximateArrivalTimestamp: 2015-12-01 04:10:58 +0000 UTC
Data: {"id":"0d669fb6-045e-4299-946c-5786037ab638","hoge":"fuga"}
...

Usage

Usage of kineis-tail:
  -f    tailing kinesis stream forever or not (like: tail -f) (default true)
  -interval duration
        seconds for waiting next GetRecords request. (default 3s)
  -iterator-type string
        iterator type. Choose from TRIM_HORIZON(default), AT_SEQUENCE_NUMBER, AT_TIMESTAMP or LATEST. (default "TRIM_HORIZON")
  -limit int
        limit records length of each GetRecords request. (default 100)
  -max-item-size int
        max byte size per item for printing. (default 4096)
  -region string
        the AWS region where your Kinesis Stream is. (default "ap-northeast-1")
  -start-time string
        timestamp to start reading. only enable when iterator type is AT_TIMESTAMP. acceptable format is YYYY-MM-DDThh:mm:ss.sssTZD (RFC3339 format)
  -stream string
        your stream name (default "your-stream")

How does it works:

  • At first, describe stream.
  • Get shard iterator.
  • GetRecords iteratively.

LICENSE

MIT

Author

Kenta Suzuki (a.k.a suzuken)