-
Notifications
You must be signed in to change notification settings - Fork 91
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature request: Pause a pipeline #256
Comments
We've actually discussed as a team the possibility of adding "pause" functionality a few times in the past, but have always concluded that the idea would be far trickier to implement than it seems. Simply stopping a pipeline is already a complicated procedure: deactivating components, disabling source components from generating new messages, pausing for quiescence, etc. It's also not entirely clear what behavior would be expected when pausing and resuming a pipeline in different scenarios, e.g., a live pipeline vs a pipeline that is replaying from a store. What are the semantics of "pausing" a pipeline? Should it "freeze the universe", halting the (virtual) clock entirely? What happens to messages that are in flight? These are questions that might be answerable, but it would require a very careful design. And it might be overkill for what you're really trying to achieve. What is the concrete use case that you have in mind exactly? There might be other, more targeted ways of achieving what you're looking for, without implementing "pause" at the pipeline level. For example, if it's primarily a replay scenario, perhaps the PsiStoreReader could be extended to allow for pausing in playback? |
After diving into the code, my initial question should have been aimed toward exposing the pipeline's To address your questions:
To stop a pipeline and finalize its components without completely closing them off (emitters, receivers) from future use.
The clock will be in the same state as it is when a pipeline completes, essentially terminated.
A new sequencing id is specified to close emitters without resetting it as a connected component in the current pipeline configuration. You are then able to finalize all nodes with the possibility to keep the current pipeline's connections. Major changesfix: defaultHandlerCaptures the initial PipelineRun events so that you can still nullify any recursive invocations. feat: Close(..., shouldResetComponent)In addition to the closing sequence id, I specify a stopping sequence id for when you finalize a component without losing its subscribers and without raising events on close. This prevents the production and receival of messages without fully closing the emitter which can be used again later. feat: OnPipelineDisposedA new handler for completing a pipeline easily distinguishes between when a For a targeted use case of these additions, I made simple changes to The changes can be found in this forked repository: master...austinbhale:psi:master. Would you all be interested in collabing over a pull request for these changes? I welcome any of your expert feedback as well :) |
I'm glad you were able to come up with a solution that works for your scenario! But let's hold off on potentially fleshing out and integrating these changes and features until we push out our next release (hopefully coming very soon). We have some changes in flight that might intersect with this in tricky ways, around resolving some remaining ambiguity and incorrect semantics we have for our pipeline start and stop times, and an individual stream's open and close times. Let us resolve those existing issues, and then come back to this. One thing I'm concerned about is how this "stop and resume" behavior would affect existing components, not just ours, but any components that others have written in their own repos. Unfortunately, the existing implicit contract between the pipeline and a component was basically that once "stop" was called, the component should not post any more messages ever, and can go ahead and start tearing down and disposing internal resources. So there are many components that would need to be rewritten and extended to allow for resuming. Perhaps we would need something like an "IResumeable" interface that allows a component to mark that it is safe to be resumed. And if a component does not implement that interface, then all bets are off. Not sure if that's the best approach, just one idea. |
Thanks also for pointing out the wrong and misleading comment in our documentation for |
To make sure we're on the same page, before we dive deeper into implications and design constraints for this feature, can you first clarify which one of the following 3 options do you need for your scenario? Option 1: Separate pipeline.RunAysnc(replayDescriptor)
// at some later point
pipeline.Stop()
// some more code here, for instance reading some final state off of some components
// at some later point
pipeline.Dispose() Decoupling Stop from Dispose would presumably enable you to read some final state off of some components (after Stop but before Dispose), and maybe organize your code in certain ways that might be not easy under the current implementation (when Stop and Dispose are entangled). But in this option once Stop is called, you wouldn't be able to call Run() again (let's stay Run would throw if you tried it) Option 2: pipeline.RunAsync(replayDescriptor)
// at some later point
pipeline.Stop()
// at some later point
pipeline.RunAsync(newReplayDescriptor)
// at some later point
pipeline.Stop()
pipeline.Dispose() This would allow you to stop a pipeline and then later be able to call Run again on it. This means however this would be an entirely new run, and would behave in the same way as if you reconstructed the whole pipeline you had before and ran the new pipeline again. This would basically save you from having to construct the pipeline all over again. Option 3: pipeline.RunAsync(replayDescriptor)
// at some later point
pipeline.Pause()
// at some later point
pipeline.Resume()
// at some later point
pipeline.Dispose() This would in essence somehow stop messages from flowing through the pipeline after I know in the long run all of the above might be desirable, but the implementation requirements for these three options are different and it would be good to first understand which one of them you need right now. In addition, it would be great to understand why you need this functionality. Can you say a bit more about why this is needed in your specific use case? (Overall, we want to be cautious about adding runtime features that increase complexity and the number of failure points/modes and would like to understand in which way the current implementation is not sufficient and whether there are any alternative solutions to the problem you have under the current implementation.) |
Thank you both for getting back on this @sandrist and @danbohus! The specific use cases for this functionality would be the following: 1. RecordImagine you are writing streams to a psi store to be played back later. Now, during this recording, the user might want to temporarily stop recording to, for example, arrange the next task and continue where the recording previously paused. Thus, they wouldn't have to create a new psi store for the next task by constructing the pipeline again. It'd be unnecessary extra work to reconstruct the same pipeline and import the new streams into the initial psi store. 2. PlaybackThe user plays a visualization of a psi store and wants to be able to pause the current state in time of the cursor reading from the store. The playback functionality shown in PsiStudio, for example, requires users to restart a store whenever they click the "stop" and "play" buttons. Additionally, most video playback software includes options to "seek" different times in a video without reloading the entire video. If the pipeline does not require complete disposal, we can simply run the pipeline again without our time-consuming pipeline creation, which ruins the immersion of our AR application. 3. StreamSimilar to recording, except you're stopping the current visualization of the application for it to be resumed later! Option 2 is ideal for giving us flexibility. My fork of psi attempts to implement this, but as @sandrist mentions, most components would have to be rewritten to adapt this behavior. Though, I hope it can help spark an idea of how it can be approached. Option 3 would also satisfy our needs if it happens to be much easier to implement in the framework's current state. If we can change the cursor's time when reading from a data store, that would be just as effective as Option 2. Then, most of the components wouldn't have to be rewritten :) Let me know if I should clarify further! Very happy to help on this issue, as I think it will be valuable for all. |
Thanks @austinbhale. This helps, but I'm still not exactly on the same page I think. Here are some follow-up clarification questions and observations: Re 1. Record "It'd be unnecessary extra work to reconstruct the same pipeline and import the new streams into the initial psi store." I'm trying to understand why reconstructing the pipeline is much work (you seem to hint to that in 2.Playback as well). Typically constructing a psi pipeline should be very short, on the order of 1 second or so. We have in our own work done AR apps where at the top level we have a StereoKit menu, the user pushes a Run button, which constructs and runs a psi pipeline, what runs until the user pushes a "Stop" button. At that point the pipeline is disposed and the "Run" button shows up again in the mixed reality view. The experience is pretty straightforward. Is there something in your case that precludes this? Does your pipeline construction take a long time for some reason? Re: "import new streams into the initial store". Can you say a bit more about why this is a requirement? Why having separate stores is not sufficient? On a related note, I assume you are familiar with how multiple successive sessions can be combined in a dataset in psi, with the ability to run batch tasks over all of them or visualize in PsiStudio? Would those facilities help in your case, or is there a reason why these runs of the pipeline need to be in the same store? Re 2. Playback "The playback functionality shown in PsiStudio, for example, requires users to restart a store whenever they click the "stop" and "play" buttons." I'm not sure I understand what you mean by "requires users to restart a store". PsiStudio does not run pipelines or play stores. When you click Play in psistudio the cursor is simply advanced and the corresponding data is shown (PsiStudio does random access to the data). "Additionally, most video playback software includes options to "seek" different times in a video without reloading the entire video". In playback mode, PsiStudio does indeed also seek to different times as it shows the data. The time we seek is the time of the cursor (which can be driven by the mouse, or by a timer that advances it when the user hits the "Play" button) "If the pipeline does not require complete disposal, we can simply run the pipeline again without our time-consuming pipeline creation, which ruins the immersion of our AR application." Can you explain more what is ruined in the immersion of the AR app? Is the problem that the pipeline creation takes a long time, or are you somehow wanting to show some parts in AR that are shown by components in the pipeline (rather than by code outside the pipeline) and when the pipeline goes down those things go down? Re 3. Stream I'm sorry but I didn't quite follow this. Can you explain more? What is the current visualization? Are we talking about a (live) PsiStudio visualization, or about various AR objects rendered by the pipeline components? |
Hi Dan, thanks for your feedback. To be clear, my examples are about an AR application using Psi. I've also noticed in our AR applications that pipeline construction takes about a second. This application uses pipelines to fuse sensor data streams together with other Psi components, save streams to a store, and retrieve streams from a store for rendering. The pipeline construction is similar to the HoloLensCaptureApp in Psi's samples repository. Thanks for the tips about multiple Psi stores and clearing up our understanding of how PsiStudio works (we thought it runs a pipeline instead of just advancing the cursor). To demonstrate the proposed pause functionality (Option 2), we have set up an example project based on the wiki (https://github.com/microsoft/psi/wiki/Brief-Introduction#3-saving-data):
In In Solution 1: Single psi storeIf we place everything in a single psi store, we would need to handle the difference in the timestamps of the streams ( Solution 2: Multiple psi storesCreating a new psi store each time we hit stop would be also fine, if the pipeline construction takes less than ~50ms, and doesn’t lead to a noticeable delay when stopping and starting the pipeline. For our scenarios, a user needs to play and pause both recording and playback quite often (like when playing a youtube video), so even a 500ms delay quickly becomes very noticeable. For that solution, we also have the question, when combining the different stores in a dataset, how the developer handles the time discrepancies between stores? Not sure atm how we do this, would we adjust the originating times so that "unpaused" streams subtract the amount of time that had been paused? I hope the code example makes our use case a bit clearer, sorry if I missed addressing any of your questions in the reply above, happy to elaborate further. |
This would be a great feature. Are there any updates on this? |
Thanks @austinbhale for all the explanations. I think I now understand better what you are trying to accomplish. It seems to me though that Option 3 might better model (though still problematic) what you are describing. Unfortunately, while doable, implementing both Option 2 and Option 3 at the runtime level are quite complex tasks with many downstream implications. Given that, and considering the semantics of what you are accomplishing vs. the semantics of originating times in the \psi runtime, my sense is than an alternative solution based on store concatenation might fit better. I will explain each of the three statements above in more detail below (my comments below are written from the perspective of aiming for a pause/resume type feature in the runtime that would work structurally in a more general case, beyond the specific use case you have presented) 1. Option 2, Option 3 and Stream SemanticsFor the sake of discussion, suppose you start the first run of the pipeline at 1pm and stop it at 2pm, then start at 3pm and stop it at 4pm. At the end of this you want to obtain a single 2-hour store. The question is what should the originating times in those messages be? It sounds like you want the originating times to start from 1pm and end at 3pm, i.e., you would want those messages to be back-to-back, as you want to create a "stitching" effect. However, this is at odds with the notion of originating times in \psi. Originating times in psi streams are meant to semantically capture the real time in the world corresponding to an event happening, like when a video frame happened, or when a piece of audio happened. In a natural implementation of option 2 therefore, when the second Now, one could change how As I mentioned in my post above, the envisioned Option 2 should behave identically an implementation that disposes and then reconstructs and re-runs the pipeline (it would just save you the time of wiring things up and initializing components, and somehow it would use the same set of log files for export). Now, with Option 3, if we do a Basically, we would no longer maintain the assumption that originating times refer to an actual time when something happened in the world. My concern with this is that this moves the design more towards the space of powerful but also dangerous (one can shoot themselves in the foot by not realizing the underlying assumptions) … Perhaps there’s an Option 3 implementation where 2. Option 2 and 3 imply big structural runtime changes, with many downstream implicationsLeaving aside for a moment the originating time semantics issue I mentioned above, there are multiple reasons why implementing either option 2 or option 3 in a robust/general way is tricky. I haven't thought through all the implications, but right off the bat some things that come to mind are:
There are probably other considerations that would need to be taken into account, so overall accomplishing a general and robust solution to this problem is quite a sizable task. At the end of the day, the originating time semantics problem I flagged above would still I think be an issue for Option 2 and 3, regardless of implementation. One could imagine an option 4, where 3. An alternative solution based on store concatenationGiven the originating time issue above, I wonder if a more appropriate solution that keeps the originating time semantics at least at the runtime level (i.e., while the pipeline is running), is one where you do collect multiple stores, and have a way to concatenate them, while shifting the time. p.s. as for pipeline construction times, it would be interesting to perf analyze that a bit to see where most of the time goes and whether that could be shortened significantly ... |
Thank you for the detailed and well-thought-out response. It's important to have a generalized solution, so the ambiguities for options 2 & 3 would be too confusing and time consuming for developers. By staying true to the \psi principle of capturing events in real time, it seems a feasible runtime feature presented here is something like an option 4, where Correct me if I'm wrong, but the current behavior of closing an Emitter is that it can never be reopened in the pipeline, as it removes all subscribers. However, a call to
In this scenario, a call to I like the sound of a tool that concatenates stores, especially if it can perform the time stitching and saving of the new store in the background. If option 4 gets implemented, I'm imagining a more flexible tool that can handle either the time stitching of separate stores or the time gaps in a single store. For the single store, a flag would be needed to indicate when the pause occurs. However, for the separate stores, no flag is needed, and one can look at the start and stop times of the two stores and see if a time stitch is possible. For example, a store that goes from 1 pm
As you say, this changing of originating times would violate the originating time's meaning, but the expectation would be there if the developer decided to use this tool. Do you think it is enough to simply use the starting and ending of pipelines as the basis for time-stitching multiple stores into a new store (as seen in the multiple store scenario above)? This implementation could also look inside individual stores for a stream of flags that indicates messages were being sent in that section of the pipeline. If a stream of pause flags does not exist, we assume it as one complete run. Otherwise, we translate the flag cycles (i.e., 1->0, 0->1) as indicators to shift the times, similar to the merging of
Also, thanks for suggesting to analyze the pipeline construction times. I've optimized our construction time, which takes a second or so to initially start cameras on the first run, but then takes less than 80 ms to record and under 20 ms to playback on successive runs! So that should be suitable for our playback needs, e.g., pause/play or moving the cursor in time. I figured a builder design pattern would be suitable for the construction of the mixed reality renderers, since you create a new instance for every pipeline. With this approach, you only need to construct the renderers once, so on every new instance of a renderer, it is simply connecting pipeline components together. |
Hello 👋 would it be possible to implement the "pausing" and "resuming" of an active pipeline?
The current way to stop an active pipeline is to call
Dispose
which "stops the pipeline and removes all connectivity (pipes)":psi/Sources/Runtime/Microsoft.Psi/Executive/Pipeline.cs
Line 680 in 8ebdc3a
Pausing would be incredibly useful for either streaming or playing back a store. For example, pausing in PsiStudio stops the pipeline and replays to the beginning once you hit play again.
I imagine much of the functionality for this
Pause
method is already contained in theStop
method:psi/Sources/Runtime/Microsoft.Psi/Executive/Pipeline.cs
Line 829 in 8ebdc3a
It needs refactoring, of course, like additional pausing and paused states, and the use of the current time as the paused originating time, but it looks feasible at first glance.
Is this feature something you all would be interested in? My current work requires pausing/playing a stream, so I'd be happy to help. Thanks!
The text was updated successfully, but these errors were encountered: