Add MessageDb feed source

[nordfjord]

No description provided.

[bartelink]

Might as well also wire this into propulsion checkpoint (see tools/propulsion.tool/program.fs)

[bartelink]

FYI I implemented an "all caught up" check in

propulsion/src/Propulsion.DynamoStore/DynamoStoreSource.fs

Line 187 in d3246c9

member x.RunUntilCaughtUp(timeout : TimeSpan, statsInterval : IntervalTimer) = task {

which is used in Lambda reactors e.g. https://github.com/jet/dotnet-templates/blob/e063a661de8154336d29eba090e62157ed6dbe6d/equinox-shipping/Watchdog.Lambda/Function.fs#L97

This could probably be generalized somewhat (it's messy to do for Cosmos, but that impl would work for esdb, sss and also mdb)

For mdb, you might be able to do a slightly more optimized thing (though the monitor here has all the context to know the instant all feed positions have caught the tail, regardless of where those checkpoints live and/or whether they've actually been committed yet)

Probably the best resolution is to do a PR to move it somewhere more general if you ever end up needing it

[nordfjord]

FYI I implemented an "all caught up" check in

Interesting! I would love to learn more about this. I hadn't considered an "all caught up" check to be possible with an ever growing sequence of events 🤔.

Looking at the code it seems to use the isTail value of the batch which I naively set to false as a constant. I've fixed that now so we assume it's the tail when we reach an empty page

What's the context you're using this in?

Based on the comments I'm gathering that you have a processor running in lambda, and you run this lambda every minute via cron. I'm guessing this is a cost optimization strategy for you since after catching up you'd be billed for the lambda essentially doing nothing? So I guess it's in a context where

1. you can afford up to a minute of delay in reactions
2. idling costs are significant

[bartelink]

Lovely work, thanks!