Use rexi:stream/1 for view backpressure #65

Merged
merged 3 commits into from Nov 20, 2012

Projects

None yet

2 participants

Owner

This uses the new rexi:stream/1 API to allow rexi workers to stream results back to the coordinator process. This is intended to reduce the sensitivity of views to RTT between nodes involved in a view response.

@davisp davisp Use rexi:stream/1 for view backpressure
This uses the new rexi:stream/1 API to allow rexi workers to stream
results back to the coordinator process. This is intended to reduce the
sensitivity of views to RTT between nodes involved in a view response.
a448312
Owner

Thinking about the upgrade strategy for something like this ... it's tricky. We could introduce the new rexi:stream interface and add new endpoints in fabric_rpc which use that interface (rather than replacing existing ones). Then in a second commit we'd modify the coordinator functions to use the new endpoints. There's a good bit of coordination involved.

Owner

@kocolosk Yeah. I'm not at all sure on the best way here. I'm currently puzzling through how to try and maintain the merge sort properly between shards but I keep needing to add a field to the collector record which worries me because of how its used in fabric_view_changes for long running requests.

kocolosk added some commits Nov 20, 2012
@kocolosk kocolosk Use fabric_rpc2 endpoints
BugzID: 13311
093b86a
@kocolosk kocolosk Preserve original implementation in fabric_rpc
Coordinators on nodes running the old release will be the only ones
hitting fabric_rpc for views, and those coordinators will be expecting
the original API.

BugzID: 13311
e1eff10
Owner

I think this is ready to go. There's a bit of new stuff in fabric_rpc regarding attachments, but fabric_doc_update is invoking the fabric_rpc endpoint directly so we'll still pick that up.

Owner
davisp commented Nov 20, 2012

LGTM

@kocolosk kocolosk merged commit b73311b into master Nov 20, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment