Merge pull request #6 from Tenzer/backfill-script

Backfill script

README.md

@@ -144,6 +144,13 @@ export DEBUG=1
 }
 ```
 
+## Backfilling data
+Traildash will only pull in data which is being added after the above has been configured, so if you have logs from before this was configured you will have to backfill that data. To make that easier you can use the `backfill.py` Python script provided to notify Traildash of the older data.
+
+The script relies on the same environment variables mentioned above, but also requires a `AWS_S3_BUCKET` variable with the name of the S3 bucket that holds your CloudTrail files. The script also requires some extra permissions than the user for CloudTrail requires, as it needs to list the files in the S3 bucket and also add items to the SQS queue.
+
+The only dependency outside of Python itself is the AWS library, Boto3. It can be installed by running `pip install boto3`.
+
 ## Development
 
 #### Contributing

backfill.py

@@ -0,0 +1,28 @@
+#!/usr/bin/env python
+
+import json
+from os import environ
+
+import boto3
+
+
+bucket = boto3.resource('s3').Bucket(environ.get('AWS_S3_BUCKET'))
+queue = boto3.resource('sqs').Queue(environ.get('AWS_SQS_URL'))
+
+
+items_queued = 0
+for item in bucket.objects.all():
+    if not item.key.endswith('.json.gz'):
+        continue
+
+    queue.send_message(
+        MessageBody=json.dumps({
+            'Message': json.dumps({
+                's3Bucket': environ.get('AWS_S3_BUCKET'),
+                's3ObjectKey': [item.key]
+            })
+        })
+    )
+    items_queued += 1
+
+print('Done! {} items were backfilled'.format(items_queued))