Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transition to Vulkan #208

Closed
16 tasks done
mitchmindtree opened this issue Nov 2, 2018 · 21 comments
Closed
16 tasks done

Transition to Vulkan #208

mitchmindtree opened this issue Nov 2, 2018 · 21 comments
Assignees
Labels
Projects

Comments

@mitchmindtree
Copy link
Member

mitchmindtree commented Nov 2, 2018

This is a follow up to #108 with some actionable steps!

The general plan is to depend on vulkano for its type-safe API and switch out every that we currently use glium for it.

This Vulkan Tutorial should be a useful reference.

  • Run some example locally to verify that a few essentials currently work on stable:
    • Basic triangle.
    • GLSL to SPIR-V.
    • Textures (images).
  • Switch from using the glutin window and glcontext to vulkano-win.
  • Expose vulkano::swapchain::Surface methods from Window.
  • Add method for enumerating vulkan physical devices.
  • Add window builder that allows for specifying the physical device to use for the window's swapchain.
  • Construct swapchain for window. Investigate whether this should be shared or unique between multiple windows.
  • Add LoopMode that only updates when the next image is ready for a specific window. Should consider making this a builder method on a window so that a unique update may be used for each if desired? Can't do this as i would require adding type params to the App.
  • Recreate Frame type using swapchain::acquire_next_image future.
  • Get LoopMode::RefreshSync run loop working with a cleared image.
  • Add in swapchain recreation on resize.
  • Multi-window support.
  • Update other LoopModes.
  • Recreate swapchains on LoopMode switch.
  • A window builder method that allows the user to construct the swapchain themselves.
  • Create vulkan based draw backend to replace the glium one. Refer to triangle example for handling buffers.
  • Switch from using the glium backend of conrod to a vulkano one.
@mitchmindtree
Copy link
Member Author

Some conditions that must be met:

  • Allow for drawing to FBOs at any time.
  • Allow for running compute shaders at any time.
  • Restrict "swapchain" (window) drawing to the view function.

@mitchmindtree
Copy link
Member Author

mitchmindtree commented Nov 5, 2018

One big difference between OpenGL and Vulkan is that OpenGL sort of implicitly queues commands for the GPU behind the scenes, whereas Vulkan is very explicit about when the GPU is ready to process the next image.

Due to this OpenGL behaviour, Nannou's current behaviour is that when view is called the user is expected to draw to all windows at once and OpenGL would implicitly process the windows when it was ready. Vulkan on the other hand exposes the exact instant that each swapchain is ready to process the next image (via the acquire_next_image future) which raises the possibility that not all swapchains might be ready at the same instant, especially when windows are spread across multiple displays or driven by different physical GPUs.

Changing the view function behaviour

Rather than expecting the user to draw to all windows at once within the view function, it could be a good idea to change the behaviour so that the user only draws to a single window each time view is called. The window for which view is called could be indicated via a window_id delivered with the Frame type. This would allow the view function to be called as soon as the swapchain for a window is ready to acquire the next image.

When drawing to multiple windows, the user would be expected to do something like this in their view function:

fn view(app: &App, model: &Model, frame: Frame) -> Frame {
    match frame.window_id() {
        id if id == model.window_a => view_a(app, model, frame),
        id if id == model.window_b => view_b(app, model, frame),
        _ => unreachable!(),
    }
}

Alternatively, we could require that a view function is passed as a builder method during the window construction process?

let window = app.new_window()
    .with_dimensions(w, h)
    .with_title("Awesome Window")
    .view(view)
    .build();

Either way, we should provide an example for this, along with an example for drawing different parts of a single framebuffer across multiple windows.

@mitchmindtree
Copy link
Member Author

The new Frame type

Seeing as the new Frame type would only contain the drawing context for a single window when it is ready for a new image, it will need the swapchain for that window and the current image that is ready to be acquired.

@mitchmindtree
Copy link
Member Author

Calls to update & The Swapchain.

I'm currently thinking a lot about how to best handle multiple windows with Vulkan.

Using Vulkan, each window must have its own Swapchain. The purpose of the swapchain is to synchronise the presentation of images with the refresh rate of the display.

In order to have the lowest possible latency, it would be useful to allow the user to specify a LoopMode that called update and then view each time vulkan indicates that the swapchain is ready to acquire a new image via acquire_next_image.

One issue that comes to mind is - how do we account for multiple windows in this case? I imagine most of the time multiple windows may be presented on the same physical device and display, meaning that the refresh rate would be the same and that the user probably only wants one call to update immediately before view is called for the first window to call acquire_next_image. One question that comes to mind is - how do you (and can you even) be sure whether the swapchains will be synchronised and which will be the first to request an image? However, we also have to consider the case where the user may have their windows spread across multiple different displays (e.g. a laptop display and one or more external monitors) - a case in which swapchains almost certainly will not be synchronised.

Do we call update once for each window? If so, does the user accept the case that update will be called at inconsistent intervals? E.g. if one display is 59.3hz and another is 90hz, the phase of the refresh rate will constantly be shifting over time, meaning sometimes there may be less than a millisecond between update calls, and sometimes a full 1/59.3 seconds. If this is considered OK, how do we determine when multiple swapchains' acquire_next_image calls will be synchronised and when they will not? Should we even bother trying to distinguish or should we just go for an update per window model?

Or perhaps instead only a single window's swapchain should drive the updates while the other swapchains simply present whatever the current state of the model happens to be when they are ready for their next respective images? If this option is worth considering, then perhaps it is worth considering not trying to synchronise with the acquire_next_image call whatsoever and instead looping at some self-timed rate (e.g. 60hz) and then let all the swapchains just present whatever the state is when acquire_next_image is ready? There are many options to consider. It would be nice to let the user choose this while also providing reasonable defaults.

LoopMode Options

These are all the possible options that come to mind for LoopModes as mentioned above:

  • One update and one view for each call to swapchain::acquire_next_image.
  • One update for each call to a single window's swapchain::acquire_next_image and a call to view for each window's swapchain::acquire_next_image.
  • An arbitrary update rate specified by the user and a call to view for each window's swapchain::acquire_next_image.

Note that the LoopMode::Wait variant should not change as it is based on user input rather than display refresh rates. E.g. when some user input is received, update is called immediately and then view is called once for each display.

@freesig
Copy link
Collaborator

freesig commented Nov 9, 2018

I don't fully understand this domain so my thoughts might be a bit misguided but I would imagine that you would not want to couple the update call to the refresh rate of the monitor. It would not be bad for example for a 90hz monitor to draw the same model twice if the scenario didn't have a reason to change the model.

I think it would be good to have a model that all monitors get in their acquire_next_image() and this model is updated by some master app loop with its own rate

@mitchmindtree
Copy link
Member Author

I would imagine that you would not want to couple the update call to the refresh rate of the monitor.

The reason I think it might be important to provide this option is in order to allow for minimizing latency as much as possible - that is the difference between the state that gets displayed and what the state actually should be at that exact instant. For example, if we have an arbitrary update rate of 60hz but the display rate is 59.6hz or something, then the update latency will drift from from near-0 ms to about 16ms in a kind of saw wave pattern over time.

I think this generally gets really important for interactive applications or games where you want your view to seem as synchronised as possible with your actions. E.g. imagine the case where the update has drifted to the point where it occurs immediately after the swapchain retrieves the next image. Seeing as user input generally isn't processed into the application state properly until an application update is called, this means that any input that occurs between the drifted update call and the next view call will not actually be displayed until not the next view call but the one after that, meaning you can pretty easily get up to two whole frames (~32ms) of input latency.

I think it would be good to have a model that all monitors get in their acquire_next_image() and this model is updated by some master app loop with its own rate

Just to clarify, acquire_next_image() is a function that is called per-swapchain and there is one swapchain per window. I don't actually know if it's possible to get the available monitor refresh rates at that point via vulkan or if the vulkan spec even guarantees that the window's swapchain will always exactly synchronise with whatever the rate of the monitor that the window happens to be on, I'm just assuming at the moment based on what I've read. It sounds like maybe what you're suggesting is similar to the idea of having one arbitrary update rate and then allowing each window to just present whatever the state is when their acquire_next_image function is ready. I think that's fine for some applications and we should definitely have a mode for that, but could become an issue for low-latency applications like mentioned above.

@freesig
Copy link
Collaborator

freesig commented Nov 9, 2018

Ah I see what you mean.
So what is the issue if they go out of sync? Is it that you are worried about them sharing the same model and one monitor might show the results of an event before another?

@freesig
Copy link
Collaborator

freesig commented Nov 9, 2018

Like if one screen gets ahead of the other it will only get the latest available state of the model.
I'm imagining that it's sort of like a pull system where events become available and then each screen pulls the events into their update loop. We would need to track which screen has already pulled an event and then once they have all received it then the event is consumed.
The same would be true for things that happen internally to your model. They would still be a type of event

@mitchmindtree
Copy link
Member Author

So what is the issue if they go out of sync?

I guess what I'm concerned about is, if you were to call update once for each swapchain's call to acquire_next_image and those swapchains were presenting at different rates, there would occasionally be moments where an update may get called twice (or more times) almost instantaneously on the occasions where those swapchains' calls to acquire_next_image all happen roughly at the same time. If that call to update was pretty expensive (like I imagine it could be for big applications) then each of the swapchains other than the one that happened to call acquire_next_image first would have to wait unnecessarily long for the update to be called again, even though they were all ready at roughly the same time and probably didn't need the extra update calls.

I'm imagining that it's sort of like a pull system

Yeah true, maybe it should be possible to be able to specify a "minimum latency" interval in order to avoid this issue? That way when acquire_next_image is called, you could check if update has not been called at all within the last minimum_latency_interval_duration or if any user input has been received since the last view occurred and if either of those conditions are met then you know you have to call update before presenting to the swapchain, otherwise you can just use the existing state. Does that kind of make sense? If so, I wonder what the best way to offer this via LoopMode would be.

@freesig
Copy link
Collaborator

freesig commented Nov 9, 2018

I was more imagining that each screen would be "subscribed" to the parts of the model that are relevant to it. Some of those parts would be shared among other screens. Each screen has it's own update that pulls in the relevant events and updates the model that it holds. Not calling one master update twice.
But yeh that could still involve duplicating work.

Or could it be more that an events are only applied once and the model is shared like

event1 on queue
screen1 calls update and this applies event1 to the model
screen2 calls update gets model with event1 already applied
screen2 calls no events, same model
event2 on queue
event3 on queue
screen2 calls update and events are applied, updated model is delivered
event4 on queue
screen1 calls update and gets model with all previous events applied plus event4 applied
screen2 gets same model that screen1 just got

I mean events in a sense more then UI events. Like you might have your physics engine sending movement events at a certain rate

@mitchmindtree
Copy link
Member Author

mitchmindtree commented Nov 9, 2018

Ok I think I get what you're saying by grouping all input and application events into a single "event".

To clarify, the reason why I am singling out user input events in particular is that those are the events that affect the feeling of "responsiveness" and low-latency to the user, as it is the point at which the user interacts with the system. The other distinction that I think is important is between events that occur from the external world (via I/O like user input, cam input, audio I/O, clocks, etc) vs events like physics updates that are generated within the application itself that the user has more control over. I'm generally lumping all the application events (physics etc) into the update function process as nannou itself can't really know exactly what application events the user might want to generate (e.g. a "todo" item was created, a pose occurred via skeleton tracking, a collision occurred, etc).

I do like the idea of allowing for an update per-window though. Perhaps similarly to allowing each window to have its own view function via a builder method on the app.new_window() call, we could also allow for passing an update function via a builder method which was called immediately when that window's swapchain's acquire_next_image function is ready? That way we don't have to try and add new LoopModes for syncing with multiple window swapchains - instead it's kind of implied that this is what you want if you pass an update function when building the window. Hmmmmmmmmm

@mitchmindtree
Copy link
Member Author

For anyone else following along, this is the new LoopMode enum I'm thinking of going with following all this discussion. The main difference with the existing one is the addition of the RefreshSync variant.

/// The mode in which the **App** is currently running the event loop and emitting `Update` events.
#[derive(Clone, Debug, PartialEq)]
pub enum LoopMode {
    /// Specifies that the application is continuously looping at a consistent rate.
    ///
    /// An application running in the **Rate** loop mode will behave as follows:
    ///
    /// 1. Poll for and collect all pending user input. `event` is then called with all application
    ///    events that have occurred.
    ///
    /// 2. `event` is called with an `Update` event.
    ///
    /// 3. Check the time and sleep for the remainder of the `update_interval` then go to 1.
    ///
    /// `view` is called at an arbitraty rate by the vulkan swapchain for each window. It uses
    /// whatever the state of the user's model happens to be at that moment in time.
    Rate {
        /// The minimum interval between emitted updates.
        update_interval: Duration,
    },

    /// Waits for user input events to occur before calling `event` with an `Update` event.
    ///
    /// This is particularly useful for low-energy GUIs that only need to update when some sort of
    /// input has occurred. The benefit of using this mode is that you don't waste CPU cycles
    /// looping or updating when you know nothing is changing in your model or view.
    Wait {
        /// The number of `update`s (and in turn `view` calls per window) that should occur since
        /// the application last received a non-`Update` event.
        updates_following_event: usize,
        /// The minimum interval between emitted updates.
        update_interval: Duration,
    },

    /// Synchronises `Update` events with requests for a new image by the swapchain for each
    /// window in order to achieve minimal latency between the state of the model and what is
    /// displayed on screen. This mode should be particularly useful for interactive applications
    /// and games where minimal latency between user input and the display image is essential.
    ///
    /// The result of using this loop mode is similar to using vsync in traditional applications.
    /// E.g. if you have one window running on a monitor with a 60hz refresh rate, your update will
    /// get called at a fairly consistent interval that is close to 60 times per second.
    ///
    /// It is worth noting that, in the case that you have more than one window and they are
    /// situated on different displays with different refresh rates, `update` will almost certainly
    /// not be called at a consistent interval. Instead, it will be called as often as necessary -
    /// if it has been longer than `minimum_latency_interval` or if some user input was received
    /// since the last `Update`. That said, each `Update` event contains the duration since the
    /// last `Update` occurred, so as long as all time-based state (like animations or physics
    /// simulations) are driven by this, the `update` interval consistency should not cause issues.
    ///
    /// ### The Swapchain
    ///
    /// The purpose of the swapchain for each window is to synchronise the presentation of images
    /// (calls to `view` in nannou) with the refresh rate of the screen. *You can learn more about
    /// the swap chain
    /// [here](https://vulkan-tutorial.com/Drawing_a_triangle/Presentation/Swap_chain).*
    RefreshSync {
        /// The minimum amount of latency that is allowed to occur between the moment at which
        /// `event` was last called with an `Update` and the moment at which `view` is called by
        /// a window's swapchain.
        minimum_latency_interval: Duration,
        /// The windows to which `Update` events should be synchronised.
        ///
        /// If this is `Some`, an `Update` will only occur for those windows that are contained
        /// within this set. This is particularly useful if you only want to synchronise your
        /// updates with one or more "main" windows and you don't mind so much about the latency
        /// for the rest.
        ///
        /// If this is `None` (the default case), `Update` events will be synchronised with *all*
        /// windows.
        windows: Option<HashSet<window::Id>>,
    },
}

@mitchmindtree
Copy link
Member Author

In an attempt to better understand the swapchain::acquire_next_image function and when the "waiting" for the display to refresh actually occurs w.r.t, i did some timing of the vulkano triangle example.

First I tried testing the time between frames on average. I tested using two different PresentModes:

  • PresentMode::Fifo - close to 16.6ms per frame, implying synchronisation with the 60hz refresh rate as expected.
  • PresentMode::Mailbox - from 100-500 microseconds per frame, implying that its giving you an image to draw to as fast as possible and just uses the most recently drawn, freely available image to actually present to the screen as documented.

This implies that Fifo would probably be well suited to LoopMode::RefreshSync, while Mailbox would be much better suited for our LoopMode::Rate. It's probably worth keeping in mind the possibility of recreating the swapchains on LoopMode change in order to optimise for this. This should be done with the consideration that we would like to allow for users to describe how swapchains should be built, and we don't want to override their preferences by recreating the swapchain ourselves.

I also found that the 16ms wait does occur during the call to swapchain::acquire_next_image.

@mitchmindtree
Copy link
Member Author

So far I've been thinking that the only approach to ensuring that the swapchain::acquire_next_image call never blocks the main thread is to use a separate thread for calling it and then sending the returned image number to the main thread so that it was able to being processing the associated image framebuffer and submit the result. This would involve quite a bit of synchronisation trickery between threads. E.g. What happens if a new window is created or an existing one destroyed? How to synchronise the recreation of the swapchain and it associated image framebuffers on events like resizing?

I just remembered that swapchain::acquire_next_image actually takes a timeout argument, and if you give a timeout of 0 the function won't block! This means we should be able to do all our swapchain handling on the main thread and not have to worry about synchronisation issues, or other issues like if you can even acquire an image from a swapchain on a non-main thread on platforms like macos.

@mitchmindtree
Copy link
Member Author

mitchmindtree commented Nov 13, 2018

The Frame type

The Frame type that is passed to the view function and returned for presentation should allow for maximum flexibility while also providing a suite of reasonable defaults for every parameter to ensure ease of use for those who don't know (or want to know) anything about Vulkan!

In order to distinguish between nannou's job and the user's, I put together a visual representation of the Vulkan dependency graph:

nannou vulkan-data flow 1

Processes that should remain under nannou's control are those that the user almost never needs or wants control over, those that require precise timing/synchronisation and those that need tight interoperation with the event loop and windowing system to avoid allowing for the creation of unexpected behaviour within nannou.

The Frame type should allow for user-submitted:

  • RenderPass - The render pass describes the destination for the output of the graphics pipeline (see below). It is essential (I think) that the render pipeline uses the same pixel format of the window's surface. It also must be initialised with the same logical device with which the surface was initialised. Thus, it would be useful if nannou could provide a helper function for creating the render pass that implicitly ensured these two correct parameters, however vulkano provides renderpass construction via the single_pass_renderpass macro, so it may not be trivial to wrap this. If we do, it might be useful to use a "session-typed" Builder in order to allow for reasonable defaults. At the very least, we should have a custom_vulkan_pipeline.rs example that demonstrates how to construct this correctly.

  • FrameBuffers - The frame buffer is the buffer to which the user will write their pixel data. There are a few tricky things to consider with the frame buffers:

    • There must be one for each image in each swapchain for each window.
    • The user should not be able to write to a FrameBuffer associated with a display outside of the view function.
    • It must be associated with the previously created RenderPass.
    • They must be re-constructed each time their associated swapchain is recreated.

    As a result of these requirements, we should maintain fairly strict control over these framebuffers. It would be nice if we could allow the user to create these using a custom render pass, however ownership over the buffers themselves should be maintained by nannou to ensure the user does not attempt to write to them or read from them outside the view function. Alternatively, we could allow for the user to create and "own" these swapchain buffers, but restrict their usage using a type-safe API e.g. requiring the view function's Frame type to be present. This would at least allow the user to create their render pass, framebuffers and graphics pipeline once and store them within their model so that they may be re-used between calls. Perhaps we could implicitly reconstruct these framebuffers when the swapchain requires recreation, and add/remove framebuffers when windows are created/destroyed, as we'll have access to the user's render pipeline anyway.

    While working on this SwapchainFramebuffer API it would be worth keeping in mind what the Draw API might look like when drawing to a framebuffer. For example, it would be nice if draw.to_frame(frame) could accept a swapchain framebuffer in the same way that it might expect any other framebuffer.

  • GraphicsPipeline - Similar to the GL "program" but much more detailed and explicit, allows for describing the fun stuff like vertex shaders, fragment shaders, etc.

  • CommandBuffer - The list of commands that are going to get executed on the GPU. Seeing as the role of view is simply to draw to a swapchain image, it may be reasonable to assume that a subset of these will be used. We probably want to avoid giving the user direct access to the vulkan command buffer builder itself as 1. many of its capabilities are irrelevant to the process of drawing to a swapchain image and 2. they may attempt to execute or submit the command buffer before returning or swap it with another built with an invalid device or queue family. Considering this, it might be best for the Frame type to wrap the command buffer and expose a subset of its methods:

    • render_pass:
      • draw
      • draw_indexed
    • copy_image
    • blit_image
    • clear_color_dimensions
    • copy_buffer
    • copy_buffer_to_image
    • copy_buffer_to_image_dimensions
    • copy_image_to_buffer
    • copy_image_to_buffer_dimensions
    • fill_buffer
    • update_buffer

    We should also ensure that it is possible to add some of the above commands, do something else, then add some more commands. In other words, the frame should not be consumed when adding commands.

    Perhaps the process of actually drawing to a Frame in a low-level manner may look like this:

    frame.add_commands()
        .render_pass(swapchain_framebuffers, clear_values).unwrap()
        .draw(graphics_pipeline, &dynamic_state, vertex_buffer,  descriptor_sets, constants).unwrap()
        //.draw_indexed(graphics_pipeline, &dynamic_state, vertex_buffer, index_buffer, descriptor_sets, constants)

    The code above adds

    .begin_render_pass(...)
    .draw(...)
    .end_render_pass(...)

    to the inner AutoCommandBufferBuilder.

Following all of this, the Frame type is returned and nannou will build and submit the command buffer!

@JoshuaBatty
Copy link
Member

This is some epic work mitch maestro! Getting excited for all this Vulkan goodness.

@mitchmindtree
Copy link
Member Author

mitchmindtree commented Nov 15, 2018

OK, so although we don't need a thread to avoid blocking on swapchain::acquire_next_image, we may still need one in order to "select" from whichever window may be ready first and to avoid prioritising one window over others. This issue is becoming a bit clearer as I begin working on the app loop for LoopMode::RefreshSync.

If we are using only the main thread, how do we check whether a window's swapchain is ready for an image without continuously looping? If we sleep for even a short amount of time, we may miss the moment at which a swapchain becomes ready. Further, how do we check the window swapchains in a way that gives even priority? E.g. if while we are iterating, one window further down the list became ready before one earlier in the list, we have no way to know this and will end up prioritising the window that is earlier in the list.

Threading Swapchain Image Selection

This brings me back to thinking that it may be necessary to use a thread per window (whether a green thread or a native thread). I can think of two options:

  • Use a native thread per window where that thread blocks on one image at a time and uses std::sync::mpsc::channel (or a crossbeam queue) to send the image to the main thread.
  • Create a futures::Stream for each window and use the tokio work-stealing runtime to automatically distribute the load across available native threads and use Stream::select to yield images from whichever swapchain is ready first.

While the native thread option would probably be fine for a couple of windows, using tokio's work-stealing runtime will probably be a safer, more scalable way of distributing the work.

State Synchronisation

While the solution appears to be threading the next-image-selection process, this also introduces some issues with synchronising the necessary state.

  • When a window is added or removed, that window's swapchain "image" stream should immediately be added/removed from the tokio runtime. Perhaps there is some way of using a Window's Drop implementation to automatically end its Stream on the tokio runtime. Similarly, the window creation process should have a way to send its swapchain to a stream on the tokio runtime.
  • The swapchain will require recreation whenever a window is resized (or a similar event occurs) and in turn will need access to the current window dimensions. This meas either 1. having access to the windows on the image-selection thread (means switching their storage in the app from RefCell to Arc) or maintaining a HashMap<window::Id, (u32, u32)> on the image-selection thread updated via a channel on Resize events on the main thread.

Further, the loop mode may change at any moment in time and it is worth considering how this would affect interaction between the main thread and the image selection thread.

The run loop is already quite complex and I can imagine threading the image selection process won't improve things, so I'm keen to properly think this out to ensure there aren't any better options before going ahead with this.

@mitchmindtree
Copy link
Member Author

Main-thread Image Selection

The previously mentioned issues with selecting the next image to process on the main thread are:

  1. How to avoid continuously looping without over-sleeping when checking each swapchain if it's ready to acquire the next image?
  2. How to ensure that all swapchains have equal priority?

Possible solutions

  1. Use the minimum_latency_interval (or the interval /2) as a sleep duration if we check all swapchains and none of them are ready? The minimum_latency_interval implies that is the minimum acceptable time to wait, so maybe it's OK if we oversleep by an amount that is within that interval? It is not ideal timing, but possibly acceptable within the API?

  2. Maybe it's OK to prioritise windows in the order in which they are created? In many cases, I imagine that swapchain images may become ready all at once (particularly if shown on the same device on the same display) and it may be useful for them to update in order.

    Perhaps the order does not matter at all, as long as after one swapchain becomes ready and its image is processed, all other swapchain images are checked directly after one at a time before beginning the loop again?

Does syncing with refresh rate require blocking?

All that said, I just realised that although we can now avoid blocking on swapchain::acquire_next_image, I'm not certain that we won't have to block somewhere else if we want to synchronise with the swapchain refresh rate. E.g. It may turn out we will have to block on the drop that occurs when re-assigning the previous frame's future (as mentioned here).

@mitchmindtree
Copy link
Member Author

mitchmindtree commented Nov 19, 2018

The meaning of LoopMode::RefreshSync

RefreshSync is a LoopMode that describes synchronising input processing and updating the user model with the rate of the display refresh rate. From my understanding, the most suited (and reliably available) vulkan present mode for this is the FIFO mode with two images (a double-buffer). In this mode, the swapchain presents one image while allowing the user to write to the other image. The swapchain swaps these images when the display is ready for a new image. I'm assuming that on a 60hz display, this will happen 60 times per second.

More specifically to nannou - RefreshSync describes a loop mode in which we want to do each of these steps at the exact moment a new image is presented to the display.

  1. Acquire the "free" image via swapchain::acquire_next_image.
  2. Process application input.
  3. Update the user model.
  4. Submit a command to the GPU for drawing to the swapchain image and then present the image.

Edit: The "Selecting Present Mode" in this tutorial has a nice thorough description on the behaviour of the vulkan present modes.

@mitchmindtree
Copy link
Member Author

mitchmindtree commented Nov 20, 2018

Ok, think I've got most of the system laid out now! Currently just trying to start with a basic clear call on the frame before getting into the graphics pipeline for the draw module.

Currently attempting to clear the frame via clear_color_image fails due to the swapchain image having an incompatible image layout. The error specifies that the image layout that fails is PresentSrc, whereas it seems we need the image to be in either General or TransferDstOptimal.

Seems to be a bug in vulkano:

mitchmindtree added a commit to mitchmindtree/nannou that referenced this issue Nov 29, 2018
This switches the windowed-graphics API from OpenGL (via glium) to
Vulkan (via vulkano). You can read more about the motivations, thoughts
and discussion behind this switch at nannou-org#208 and nannou-org#108.

There are still a few items left to complete before this PR is ready:

- [ ] Fix bug where view seems to be vertically flipped (probably in
  vertex mapping process).
- [ ] Fix bug where clearing an image that is then multisampled does not
  work. vulkano-rs/vulkano#1123
- [ ] Add depth attachment to draw renderpass.
- [ ] Fix bug where alpha channels always seem opaque.
- [ ] Update old loop modes for changes in the application loop.
- [ ] Merge conrod vulkan backend and switch to it.
- [ ] Merge and publish vulkano-rs/vulkano#1117 or related fix so we can
  switch to crates.io dep.

Closes nannou-org#208.
Closes nannou-org#108.
@freesig freesig added this to To do in v0.9 Dec 23, 2018
@mitchmindtree
Copy link
Member Author

It's been a long journey, but all these boxes are ticked! There are still a few small bug fixes and some macos build automation to land but they are mostly upstream issues. Otherwise, v0.9 should be just about Vulkan ready!

v0.9 automation moved this from To do to Done Jan 2, 2019
mitchmindtree added a commit that referenced this issue Mar 30, 2019
This switches the windowed-graphics API from OpenGL (via glium) to
Vulkan (via vulkano). You can read more about the motivations, thoughts
and discussion behind this switch at #208 and #108.

There are still a few items left to complete before this PR is ready:

- [ ] Fix bug where view seems to be vertically flipped (probably in
  vertex mapping process).
- [ ] Fix bug where clearing an image that is then multisampled does not
  work. vulkano-rs/vulkano#1123
- [ ] Add depth attachment to draw renderpass.
- [ ] Fix bug where alpha channels always seem opaque.
- [ ] Update old loop modes for changes in the application loop.
- [ ] Merge conrod vulkan backend and switch to it.
- [ ] Merge and publish vulkano-rs/vulkano#1117 or related fix so we can
  switch to crates.io dep.

Closes #208.
Closes #108.
mitchmindtree added a commit that referenced this issue May 11, 2019
This switches the windowed-graphics API from OpenGL (via glium) to
Vulkan (via vulkano). You can read more about the motivations, thoughts
and discussion behind this switch at #208 and #108.

There are still a few items left to complete before this PR is ready:

- [ ] Fix bug where view seems to be vertically flipped (probably in
  vertex mapping process).
- [ ] Fix bug where clearing an image that is then multisampled does not
  work. vulkano-rs/vulkano#1123
- [ ] Add depth attachment to draw renderpass.
- [ ] Fix bug where alpha channels always seem opaque.
- [ ] Update old loop modes for changes in the application loop.
- [ ] Merge conrod vulkan backend and switch to it.
- [ ] Merge and publish vulkano-rs/vulkano#1117 or related fix so we can
  switch to crates.io dep.

Closes #208.
Closes #108.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
No open projects
v0.9
  
Done
Development

No branches or pull requests

3 participants