Skip to content

Commit

Permalink
Merge branch 'master' of http://github.com/frankmcsherry/blog
Browse files Browse the repository at this point in the history
  • Loading branch information
frankmcsherry committed Apr 1, 2019
2 parents 802c6f5 + fdaa9bd commit e3989f9
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 3 deletions.
2 changes: 1 addition & 1 deletion posts/2015-12-19.md
Expand Up @@ -277,7 +277,7 @@ One of the most appealing characteristics of this protocol is that it is totally

A slightly lazy worker `D` who doesn't receive any messages can just receive the three progress updates and confirm that they cancel out and that it shouldn't expect anything more; it doesn't need to actively participate unless it does some work. Workers `A`, `B`, and `C` don't have to wait on `D`, who can be asleep if it really wants (or maybe working on something else). Nor do the worker explicitly syncronize; they can't fire notifications until they learn that work is done, but they can work on whatever else they like in the meantime.

The downside of the protocol is the broadcasting. In Naiad we started with a very simple version of the protocol, and found that there were *lots* of progress updates being sent around. We introduced several optimizations in the paper, including hierarchical aggregation and some clever buffering. The Rust codebase doesn't seem to suffer from this as much, possibly due to the abstraction boundaries suppressing spurious changes, the it hasn't been exercised at the same scale, either.
The downside of the protocol is the broadcasting. In Naiad we started with a very simple version of the protocol, and found that there were *lots* of progress updates being sent around. We introduced several optimizations in the paper, including hierarchical aggregation and some clever buffering. The Rust codebase doesn't seem to suffer from this as much, possibly due to the abstraction boundaries suppressing spurious changes, it hasn't been exercised at the same scale, either.

## Conclusions

Expand Down
4 changes: 2 additions & 2 deletions posts/2019-02-09.md
Expand Up @@ -82,7 +82,7 @@ This is also a minor variation, where instead of *adding* path lengths as we go,
.map(|(src,(dst,wid),max)|
(dst, ::std::cmp::min(wid, max))
)
.concat(&wdith)
.concat(&width)
.reduce(|key, input, output| {
// push maximum width (input sorted).
output.push((*input.last().unwrap().0, 1))
Expand Down Expand Up @@ -359,4 +359,4 @@ In both cases we are only adding edges, but in the second case we have specializ

When you find yourself in a situation where you only need to add, or otherwise *improve*, your input data, rather than retract options, you may be able to get a leaner implementation by thinking through whether your updates look like a monoid!

It's going to depend on your computation, on your data, and possibly on a bit of thinking, but you have more options now with differential dataflow than ever before!
It's going to depend on your computation, on your data, and possibly on a bit of thinking, but you have more options now with differential dataflow than ever before!

0 comments on commit e3989f9

Please sign in to comment.