Audit log missing entries

[lyricnz]

The audit log from a run last night appears to be missing entries:

• revolver ran at 6pm AEST (0600Z), and stopped a bunch of EC2 in one account, which can be seen from audit-console log.

• but these are missing completely from the audit.log in S3 at the same time

Suspect race-condition in appending to the S3 audit log? Multiple accounts are running at the same time, and the S3 file writer doesn't have native/atomic file append, so fakes this by reading contents + appending + writing, which could result in lost-writes.

[lyricnz]

Technically this can happen with files too, not just S3, since fs.writeFile() is not atomic.

[lyricnz]

@abukharov thoughts?

• a mutex (on output filename)
• buffering all the output and writing at the end
• saying "just don't do that" and making users configure unique output files for each account+region? Which makes for ugly+fragmented audit logs:
  • audit.aws-whatever-account-name-nonprod.ap-southeast-2.csv
  • audit.aws-whatever-account-name-nonprod.ap-southeast-4.csv
  • audit.aws-whatever-account-name-nonprod.ap-somewhere-2.csv
  • ...

[lyricnz]

For emergency fix, I split the files to be unique by account+region, but this makes for rather messy output directory/S3 bucket.

[lyricnz]

As discussed with Alex: rather than implementing any locking/mutex in AccountRevolver, the logging could be deferred until all the accounts are completed, then merged, and emitted at the global scope. I'm going to delay implementing this for now, as the workaround (per-account output files) is OK for now.