Add a warning to S3 storage docs about how users experience S3 down time

[yanokwa]

An insider reported that..

I am self hosting and had a problem with the S3 storage provider - it went offline on Sunday 12th April and I was in XXXXXXX running a training course on how to use ODK, without a server that could download forms to new devices. So, as a backup strategy, I signed up for a month's subscription and intended to upload the forms to the ODK Cloud server and use them that way (without entity access of course!).

Fortunately, we managed to find enough devices that already had the form definition downloaded so were able to work using the self-hosted server on the first day of training. Coming back from a day out in the field on Thursday 16th I found that the S3 storage was back online and the Central server was working as expected. I have just got back to the office and I'm trying to untangle all the issues that arose while I was away, including an enquiry to the S3 provider (XXXXXX) asking for an explanation of 4 days downtime on their platform.

So that experience highlights a weak spot for me... S3 storage is great, except when it isn't. In the scenario where a Central Server uses S3 storage, and the S3 link is down, although the server still accepts submissions, it appears that form definitions cannot be downloaded (where they have any stored attachments) nor can submissions that have attachments that have already been transferred to S3 (more obviously!). I guess it is a very rare occurrence, but I just got hit by it. There's not a simple solution here as far as I can see, except perhaps to be able to reconfigure the Central Server on demand to use a different S3 bucket (and to have a spare bucket with another provider that has not gone down, with a copy of the original S3 bucket). Another (complicated) option might be to hold all form definition files on the Central Server so that it operates independently of S3 for distributing data to devices (or loading forms in Enketo) - it already seems to be robust enough to cope with S3 going down. The good news is that I haven't lost any data (which is sort of the point of S3 I think, and I double checked my Central backups were running hard!)

It'd be nice if we warned people in the docs about what happens if S3 is down.