Retry all with large number of messages not performant at all

[johnsimons]

I have just filled SC with 232,000 error messages and then issue a retry all via REST.

The code has been running for 20mins!

I don't think the current implementation works if there are a lot of errors to be retried.

We currently using the Stream api, what other options do we have?

//cc @andreasohlund @synhershko

[dannycohen]

@johnsimons -
What's the estimated retry rate ? (msg/sec)

[synhershko]

Have you waited for indexing to complete? maybe related to #222?

Other options would be to load all docs using LoadStartingWith - that would at least allow us to narrow down the bottleneck (could be something else and not RavenDB)

[johnsimons]

What's the estimated retry rate ? (msg/sec)

I haven't measure it, but my retry all still hasn't finished

[johnsimons]

I actually think we need to take a snapshot and then page through

[dannycohen]

@johnsimons / @indualagarsamy -
Related issue: we need to add paging support for Failed Messages in ServicePulse (or otherwise limit the number of displsyed failed messages). This already exists in SI so its not an issue there.

@indualagarsamy - Do you want to open the requirement GH Issue ?

[johnsimons]

@dannycohen SP failed messages screen already does paging.
We do it as you scroll down, no buttons.

[dannycohen]

@johnsimons - so load on-demand ? Cool. thanks.

[johnsimons]

@andreasohlund should we split the transaction ?

[johnsimons]

@andreasohlund just a reminder for you to do the test where you actually leave the endpoint on

[johnsimons]

@andreasohlund ping!

[johnsimons]

Not much we can do here, the sagas are just slow