DB may store duplicate outbound messages

[timwis]

Just thought of a scenario:

1. User sends an outbound message
2. Client generates a random ID for it, stores it in pouch, which syncs to couch
3. Server sends the message to twilio, which generates a provider id
4. Server stores the provider id in the message and is done
5. Later, the server restarts and fetches "messages since most recent message in DB" which happens to be the outbound message described above
6. Twilio gives an array of messages back, including the outbound message generated above
7. Server keys messages by their provider id and ends up creating a duplicate record for the outbound message described above in couch

This is a significant architectural issue

[timwis]

Posted on stack overflow

[timwis]

In progress at #33

[timwis]

15:27 T<timwis> (3) i can query twilio's api for messages >= a particular *date*, but not a particular date*time*. the result is the response often includes messages that are already in couchdb
15:32 R<+rnewson> (1) that's fine, and you said it already I just didn't read it.
15:42 R<+rnewson> ok, so if when you do (3) you were posting to a view (with the "keys" request body, listing the twilio id's), you'll get back rows for all the ones that existed before the crash so long as you got the twilio id back and updated the doc, and you'll have misses (i.e, no rows) for some of them
15:43 and the latter falls into two categories. 1) brand new messages, which you'd just make a new doc for (with a server-assigned _id) and 2) messages you made a doc for previously but crashed before seeing the twilio id or saving the update.
15:43 I'm breaking for food, biab
15:45 T<timwis> Tim interesting, so you're saying i could create a view that lists messages keyed by their twilio id. and at server start, i'd fetch "new" messages from twilio, then loop through each one and try to find the relevant doc _id from the view, and only insert the missing ones.
15:47 (well, i may want to insert all of them: if an outbound message is dispatched, its status is "pending." if the server shuts down before the StatusCallback boomerangs, it won't get stored in couchdb. but at server start, my fetch of recent messages would include that one, and i'd want to update its status in couch)
16:01 R<+rnewson> timwis: basically, yes. I would say you'd really call the view once with a request body like {"keys": ["twilioid1","twilioid2"...]}
16:01 but that's conceptually the same as what you said, it's just fewer requests
16:01 T<timwis> Tim oh, good idea
16:02 R<+rnewson> that "keys" post will return in the same order too
16:02 T<timwis> Tim so i'd call the view to get any existing documents, insert the _id and _rev into the messages where there is one, and create a new _id if there isn't one, then bulk insert them into the db
16:03 R<+rnewson> right.