Skip to content

2.0.0 - Adding surge protection for AWS
Compare
Choose a tag to compare
@benjben benjben released this 15 Sep 10:24
· 212 commits to master since this release
2.0.0
5ad57ca

Surge protection

While PubSub is auto-scalable, Kinesis is not, and when Kinesis can't keep up with the throughput (in case of extreme traffic spikes for instance), events accumulate in collector memory for later (infinite) retries, which can lead to OutOfMemory errors.

To address this potential issue, it is now possible to specify a SQS buffer where to write events to in case of a problem when writing to Kinesis (e.g. maximum throughput exceeded).

To activate this feature, create the SQS queues (one for enriched events and one for bad rows) and populate these lines in the configuration of the collector. Please note that writing to SQS will be retried maximum 10 times and that the size limit of an event drops to 192kb.

To send events back from SQS to Kinesis, this app can be used. It is available on Docker Hub. To run it, specify SQS_QUEUE, KINESIS_STREAM_NAME and SENTRY_DSN environment variables. The writing to Kinesis is performed with back-pressure to not overwhelm it.

Changelog

  • Disable default redirect (#64)
  • Bump vulnerable libs (#56)
  • Implement surge protection (#57)
  • Add test for extracting a URL-encoded schema from the querystring (#60)
  • Add snyk monitor (#52)
  • Add DockerHub credentials to .travis.yml (#49)
  • Add travis integration (#59)

Breaking change

  • Disable default redirect (#64)
Assets 2
Loading