Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow import operator to shut down at frontier #238

Open
frankmcsherry opened this issue Dec 3, 2019 · 0 comments
Open

Allow import operator to shut down at frontier #238

frankmcsherry opened this issue Dec 3, 2019 · 0 comments

Comments

@frankmcsherry
Copy link
Member

Some uses of differential dataflow aim to observe an output at a specific time and then walk away from the dataflow. Such a case must currently start a streaming dataflow, await the visibility of results, and then pro-actively shut down the input source.

It seems that an import operator should be able to take instruction about when to drop its capability and cease propagating batches. For example, one could supply a vector of timestamps with the intended semantics that the output must be correct for any time less or equal an element of the vector. Once the trace's upper frontier is not less or equal to any element of the vector it can drop its capability. Batches can also be filtered, unless this breaks everything horribly, by looking at the differences between its upper and lower frontiers and seeing if any timestamps are greater or lower than this bound.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant