fix(archiver): ensure items are returned to their list if any failure occurs

[ljgray]

Issue: if an error occurs after a dataset/state has been popped from a list but before it has been added to the chime database (such as a connection error with the database), that dataset/state will be lost from the list and won't ever be added to the database.

This just wraps the entire process of popping a dataset/state from a list and adding it to the chime database in a try-except block to make sure the dataset/state gets pushed back to the list if any sort of error occurs. We could consider rewriting this logic, but I think this is a minimal fix for it which shouldn't really affect any other behaviour

[ketiltrout]

I'd like a log message (thought it doesn't have to be the one I suggest) to help us detect when we actually hit this except block. Other than that, I think it's fine.

[ljgray]

I've updated it to log any uncaught exceptions, but also to separately wrap the pushback in a try-except which will dump the comet request string to the logs if the pushback fails. This might help us recover manually if there's a redis issue, although it's hard to say how useful that will be depending on what's actually happened

[ketiltrout]

Even better.

[ljgray]

I'll merge now, but when should we deploy?

[ketiltrout]

I think it's fairly safe to do so at any time because it doesn't affect the running instrument, but it's @mondana 's call.

[jrs65]

@ljgray Longer term it's probably more robust to do something like this:

• Add an extra list for states being processed (probably should end up with at most one entry in it)
• Have the main loop first process states in this list (i.e. add them to the database). One the state is processed successfully use LREM to remove it from the processing list.
• After this has been done have the main loop use BLMOVE to wait for a new list item and move it to the processing list in a single call.

That should be robust to connection failures both for MySQL and Redis.