New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WSI extension wish: user/custom surface & swapchains #205

Closed
cheery opened this Issue May 3, 2016 · 31 comments

Comments

Projects
None yet
8 participants
@cheery

cheery commented May 3, 2016

reddit link

There's already a WSI extension that lets you obtain a physical display and render to it. That's nice and I hope it'd be supported on all cards eventually. But I feel this is missing too. I wish we had this kind of extension to WSI: VK_KHR_user_surface

Defines a vkCreateSwapchainReaderKHR - At this point you put in what kind of surfaces you accept and the PID of the process that should be allowed to create a surface from the handle, when you want to be sure that only one process can write to it.

Defines a vkGetSwapchainReaderHandleKHR(VkSwapchainReaderHandleKHR, VkSize length, void* handle) that can be used to obtain a handle to the swapchain readpoint. The user of this API would take care of passing the handle to the writing process.

Defines vkCreateUserSurfaceKHR - The writing process would pass the handle here.

Defines vkGetSwapchainReaderImageKHR - that could be used by the reading process to obtain a virtual image that represents the currently bound presented image.

Additionally you'd need some synchronization mechanisms to sync the swapchains, which would make it bit more complex API than presented here.

I hoped we had this for few reasons:

  • It would allow applications to compose image streams that come from other applications in portable manner. Such situation might happen on software such as video streaming apps and VR drivers. In combination with VK_KHR_display* extensions it could drop lot of issues in making these things portable.
  • It would let ordinary people to write desktop compositors that work on the hardware and on top of each other without having to ask support from graphics card vendors for their app. Chances are this would increase the amount of decent desktop compositors while it increases amount of nondecent ones too.

Nevertheless and overall, it'd be cool to have some method to have applications communicate without having to download the data from GPU just to upload it back to GPU. I propose some kind of link-passing like here would be appropriate.

@ratchetfreak

This comment has been minimized.

Show comment
Hide comment
@ratchetfreak

ratchetfreak May 3, 2016

Sounds like what you really want is to share vkImages across several applications where one writes and the other reads with some syncing options.

There is no need for an entire swapchain for that. Just the ability to create a shared device allocation is enough.

This can be done with adding an extension struct to vkAllocateMemory which needs a ID of the memory block allocated/referenced and an enum that describes what to do if the block already exists (like exists for creating files).

Then using out of band synchronization will be enough to ensure no concurrent read and writes will occur.

ratchetfreak commented May 3, 2016

Sounds like what you really want is to share vkImages across several applications where one writes and the other reads with some syncing options.

There is no need for an entire swapchain for that. Just the ability to create a shared device allocation is enough.

This can be done with adding an extension struct to vkAllocateMemory which needs a ID of the memory block allocated/referenced and an enum that describes what to do if the block already exists (like exists for creating files).

Then using out of band synchronization will be enough to ensure no concurrent read and writes will occur.

@cheery

This comment has been minimized.

Show comment
Hide comment
@cheery

cheery May 3, 2016

That sounds it could also satisfy these needs. I don't know what out of band synchronization means though.

cheery commented May 3, 2016

That sounds it could also satisfy these needs. I don't know what out of band synchronization means though.

@ratchetfreak

This comment has been minimized.

Show comment
Hide comment
@ratchetfreak

ratchetfreak May 3, 2016

Using interprocess communication of some description (like a file pipe or shared memory) to coordinate the accesses to the shared VkDeviceMemory.

ratchetfreak commented May 3, 2016

Using interprocess communication of some description (like a file pipe or shared memory) to coordinate the accesses to the shared VkDeviceMemory.

@Themaister

This comment has been minimized.

Show comment
Hide comment
@Themaister

Themaister May 3, 2016

I think shareable VkDeviceMemory and VkSemaphores would be enough to allow implementing inter-process compositors well. Technically, VkSemaphores could be replaced with CPU-side waits and IPC, but it seems inoptimal to not allow device-side synchronization for compositors.

Themaister commented May 3, 2016

I think shareable VkDeviceMemory and VkSemaphores would be enough to allow implementing inter-process compositors well. Technically, VkSemaphores could be replaced with CPU-side waits and IPC, but it seems inoptimal to not allow device-side synchronization for compositors.

@cheery

This comment has been minimized.

Show comment
Hide comment
@cheery

cheery May 8, 2016

I wrote a blog post to raise awareness of how important this is: http://boxbase.org/entries/2016/may/9/linux-on-the-desktop-missing-piece/

If there's anything else I can do for this, I would like to know that.

cheery commented May 8, 2016

I wrote a blog post to raise awareness of how important this is: http://boxbase.org/entries/2016/may/9/linux-on-the-desktop-missing-piece/

If there's anything else I can do for this, I would like to know that.

@krOoze

This comment has been minimized.

Show comment
Hide comment
@krOoze

krOoze May 9, 2016

Contributor

@cheery Well, Vulkan WSI is not really a full-fledged compositor nor IPC library. It tries to do the absolute minimum to integrate Vulkan with the OS. So maybe barking up the wrong tree here?

Display WSI is clearly ment only for devices/OS without windowing system. In user-space, I think you can't even have such direct access to the monitor.

Well, why not, if it's possible, allow users to have the display access. But again, Vulkan is GPU abstraction API, not monitor API nor compositor API nor window creation API. So best to do it outside Vulkan. Vulkan only needs to know/abstract where and how to hand over its Images. Anything else would be feature creep, no?

BTW, what happens if you supply same Window to multiple Surfaces (possibly from multiple processes). Doesn't seem to be prohibited.

For the matter of inter-Device and inter-API(e.g. interop with OGL and DirectX) memory mapping, by all means, allow skipping the middle-man(host-memory), if the GPUs support more efficient copying. As for inter-GPU device-local memory sharing: does any existing devices even support that (without secretly copying under the hood anyway)?

Contributor

krOoze commented May 9, 2016

@cheery Well, Vulkan WSI is not really a full-fledged compositor nor IPC library. It tries to do the absolute minimum to integrate Vulkan with the OS. So maybe barking up the wrong tree here?

Display WSI is clearly ment only for devices/OS without windowing system. In user-space, I think you can't even have such direct access to the monitor.

Well, why not, if it's possible, allow users to have the display access. But again, Vulkan is GPU abstraction API, not monitor API nor compositor API nor window creation API. So best to do it outside Vulkan. Vulkan only needs to know/abstract where and how to hand over its Images. Anything else would be feature creep, no?

BTW, what happens if you supply same Window to multiple Surfaces (possibly from multiple processes). Doesn't seem to be prohibited.

For the matter of inter-Device and inter-API(e.g. interop with OGL and DirectX) memory mapping, by all means, allow skipping the middle-man(host-memory), if the GPUs support more efficient copying. As for inter-GPU device-local memory sharing: does any existing devices even support that (without secretly copying under the hood anyway)?

@cheery

This comment has been minimized.

Show comment
Hide comment
@cheery

cheery May 9, 2016

@krOoze This is platform dependent. On Linux, X11, wayland works as userspace software. On windows they recently added direct driver mode that seems to allow oculus drivers to grab a HMD display. When it's available, implementation that works on every platform would be worthwhile.

Vulkan is GPU abstraction & compute API indeed. The display ports extrude from the end of a GPU card. Vulkan WSI already comes with the VK_KHR_display extension. So implementing that extension on desktop would very likely not be feature creep.

cheery commented May 9, 2016

@krOoze This is platform dependent. On Linux, X11, wayland works as userspace software. On windows they recently added direct driver mode that seems to allow oculus drivers to grab a HMD display. When it's available, implementation that works on every platform would be worthwhile.

Vulkan is GPU abstraction & compute API indeed. The display ports extrude from the end of a GPU card. Vulkan WSI already comes with the VK_KHR_display extension. So implementing that extension on desktop would very likely not be feature creep.

@krOoze

This comment has been minimized.

Show comment
Hide comment
@krOoze

krOoze May 9, 2016

Contributor

Well yeah, it's platform dependent, that's why it's an extension. Conceivably, some platforms may have neither surface nor even display .

Well the X11 or Wayland or whatever is already running won't probably yield its exclusive use of the Display. Anyway, if user has windowing system running on a given monitor, he probably likes the software to honor it, not override it. It's a bit like wanting to run real-mode apps after you already booted into protected-mode OS.

allow oculus drivers to grab a HMD display

Well they need to make it accessible only through Oculus SDK (for post-processing or possibly vendor lock-in). They may need new OculusSurface extension for Vulkan...

The display ports extrude from the end of a GPU card.

You are making assumptions. Not on my notebook. And there is joystick port extruding from my sound card. Should it be part of all sound APIs? Besides GPU is not the same as a card.

Contributor

krOoze commented May 9, 2016

Well yeah, it's platform dependent, that's why it's an extension. Conceivably, some platforms may have neither surface nor even display .

Well the X11 or Wayland or whatever is already running won't probably yield its exclusive use of the Display. Anyway, if user has windowing system running on a given monitor, he probably likes the software to honor it, not override it. It's a bit like wanting to run real-mode apps after you already booted into protected-mode OS.

allow oculus drivers to grab a HMD display

Well they need to make it accessible only through Oculus SDK (for post-processing or possibly vendor lock-in). They may need new OculusSurface extension for Vulkan...

The display ports extrude from the end of a GPU card.

You are making assumptions. Not on my notebook. And there is joystick port extruding from my sound card. Should it be part of all sound APIs? Besides GPU is not the same as a card.

@cheery

This comment has been minimized.

Show comment
Hide comment
@cheery

cheery May 9, 2016

With Vulkan it's completely optional to use WSI. You could also render to a file or into a web stream.

VK_KHR_display doesn't need to allow access on displays that are already in use. Windowing systems tend to grab every displays that are attached, but this can be controlled and you can also run without a windowing system.

One of the merits of OpenGL and Vulkan is to avoid vendor-dependent APIs that could potentially form vendor-lock-ins. There will be bunch of HMD manufacturers beside Oculus. Do you propose every single one of them to play together with every single graphics card vendor to provide their own SDK that does the same thing?

The joystick port in your sound card originates from soundcard vendor decision to include it for boosting sales. Logically sound card can be observed as two devices on one card. It even makes sense technically because the game port holds 4 analog inputs that can be relatively easily included to hardware that got to deal with analog signals in the first place. If it's a newer sound card, it's purely for legacy reasons though, but still represents a separate device on the card.

I pointed out where the display ports are because it's very nontechnical way to point out these things naturally go together. It's easy to tear a hole when it technically is more complex than that.

cheery commented May 9, 2016

With Vulkan it's completely optional to use WSI. You could also render to a file or into a web stream.

VK_KHR_display doesn't need to allow access on displays that are already in use. Windowing systems tend to grab every displays that are attached, but this can be controlled and you can also run without a windowing system.

One of the merits of OpenGL and Vulkan is to avoid vendor-dependent APIs that could potentially form vendor-lock-ins. There will be bunch of HMD manufacturers beside Oculus. Do you propose every single one of them to play together with every single graphics card vendor to provide their own SDK that does the same thing?

The joystick port in your sound card originates from soundcard vendor decision to include it for boosting sales. Logically sound card can be observed as two devices on one card. It even makes sense technically because the game port holds 4 analog inputs that can be relatively easily included to hardware that got to deal with analog signals in the first place. If it's a newer sound card, it's purely for legacy reasons though, but still represents a separate device on the card.

I pointed out where the display ports are because it's very nontechnical way to point out these things naturally go together. It's easy to tear a hole when it technically is more complex than that.

@krOoze

This comment has been minimized.

Show comment
Hide comment
@krOoze

krOoze May 9, 2016

Contributor

Yes, I guess, why not support that. In the worst case, it would enumerate zero Displays. But that's driver-makers' decision.

One of the merits of OpenGL and Vulkan is to avoid vendor-dependent APIs that could potentially form vendor-lock-ins.

I don't think, that is the case. At least not outside its singular purpose. AIS it does the absolute minimum outside its scope to get things done. It does not create multiplatform windows either, does not control/monitor fan speeds, power-states, frequencies, gpu usage %, gpu memory usage, temperature, display brightness + contrast, conector(DVI, HDMI, DP) settings, have no multiplatform threads, could continue forever...

On the other hand it does not prevent anyone from making extensions. So could happen...
It will probably need new OculusSurface (which will take Oculus SDK swapchain). I would gladly yell at them to play together, but I think they will not with HTC. You can't do "specification by comitee" and force anyone to use it. BTW the WSI swapchain already has stereo support.

Contributor

krOoze commented May 9, 2016

Yes, I guess, why not support that. In the worst case, it would enumerate zero Displays. But that's driver-makers' decision.

One of the merits of OpenGL and Vulkan is to avoid vendor-dependent APIs that could potentially form vendor-lock-ins.

I don't think, that is the case. At least not outside its singular purpose. AIS it does the absolute minimum outside its scope to get things done. It does not create multiplatform windows either, does not control/monitor fan speeds, power-states, frequencies, gpu usage %, gpu memory usage, temperature, display brightness + contrast, conector(DVI, HDMI, DP) settings, have no multiplatform threads, could continue forever...

On the other hand it does not prevent anyone from making extensions. So could happen...
It will probably need new OculusSurface (which will take Oculus SDK swapchain). I would gladly yell at them to play together, but I think they will not with HTC. You can't do "specification by comitee" and force anyone to use it. BTW the WSI swapchain already has stereo support.

@cheery

This comment has been minimized.

Show comment
Hide comment
@cheery

cheery May 9, 2016

You can continue that list of things Vulkan doesn't provide with custom swapchains or inter-process shareable GPU resources. Correctly all those things you mentioned do not need to be in Vulkan. Some of them shouldn't be either. Consider if you can achieve as good results by excluding them away from Vulkan versus having them in.

The HMD vendors will eventually converge to something and they shouldn't be forced to that too early by outside pressure. But they face similar limitations here as everyone else and their decision making is effected by those limitations.

To get a Vulkan extension requires that you register one with Khronos and simultaneously deal with graphics card vendors to get them implement the extension too. Otherwise you will carry out the task in some non-cross-platform way.

If you had a way to establish custom swapchain across any two application, it could be also used by HMD vendors and they all could provide good, cross-platform user experience for their devices because not having to negotiate for even the most basic needs they have.

cheery commented May 9, 2016

You can continue that list of things Vulkan doesn't provide with custom swapchains or inter-process shareable GPU resources. Correctly all those things you mentioned do not need to be in Vulkan. Some of them shouldn't be either. Consider if you can achieve as good results by excluding them away from Vulkan versus having them in.

The HMD vendors will eventually converge to something and they shouldn't be forced to that too early by outside pressure. But they face similar limitations here as everyone else and their decision making is effected by those limitations.

To get a Vulkan extension requires that you register one with Khronos and simultaneously deal with graphics card vendors to get them implement the extension too. Otherwise you will carry out the task in some non-cross-platform way.

If you had a way to establish custom swapchain across any two application, it could be also used by HMD vendors and they all could provide good, cross-platform user experience for their devices because not having to negotiate for even the most basic needs they have.

@cubanismo

This comment has been minimized.

Show comment
Hide comment
@cubanismo

cubanismo May 18, 2016

The Vulkan group within Khronos is interested in the use cases covered in this proposal as well, and we are currently reviewing various extensions to the Vulkan API to address them.

cubanismo commented May 18, 2016

The Vulkan group within Khronos is interested in the use cases covered in this proposal as well, and we are currently reviewing various extensions to the Vulkan API to address them.

@ratchetfreak

This comment has been minimized.

Show comment
Hide comment
@ratchetfreak

ratchetfreak May 18, 2016

Allowing to share a vkMemory allocation will help a screen-capturing layer to isolate the application being captured from the capturing itself. So if the game crashes the capturing software is still running and can shut down cleanly and not leave a corrupt capture file.

ratchetfreak commented May 18, 2016

Allowing to share a vkMemory allocation will help a screen-capturing layer to isolate the application being captured from the capturing itself. So if the game crashes the capturing software is still running and can shut down cleanly and not leave a corrupt capture file.

@daurnimator

This comment has been minimized.

Show comment
Hide comment
@daurnimator

daurnimator Feb 15, 2017

Any progress/updates?

daurnimator commented Feb 15, 2017

Any progress/updates?

@cubanismo

This comment has been minimized.

Show comment
Hide comment
@cubanismo

cubanismo Mar 1, 2017

Yes, some of this work is finally public. Please take a look at the recently published VK_KHX_external_[memory,semaphore] extensions. One goal of these is to fullfill this usage case. Using those, it should be possible to implement a Vulkan implicit layer that exposes a new WSI platform based entirely on Vulkan primitives. For example, a render-to-texture/VkImage test harness surface/swapchain could be built, or a cross-process client->compositor windowing system could be implemented entirely in Vulkan. VKFence support would be required and is currently missing, so we're aware there's a bit more work to do.

In the meantime, feedback on the KHX external memory and semaphore extensions would be greatly appreciated, especially if someone wants to try to quickly code up some of the ideas here and validate the extensions are sufficient to enable them.

cubanismo commented Mar 1, 2017

Yes, some of this work is finally public. Please take a look at the recently published VK_KHX_external_[memory,semaphore] extensions. One goal of these is to fullfill this usage case. Using those, it should be possible to implement a Vulkan implicit layer that exposes a new WSI platform based entirely on Vulkan primitives. For example, a render-to-texture/VkImage test harness surface/swapchain could be built, or a cross-process client->compositor windowing system could be implemented entirely in Vulkan. VKFence support would be required and is currently missing, so we're aware there's a bit more work to do.

In the meantime, feedback on the KHX external memory and semaphore extensions would be greatly appreciated, especially if someone wants to try to quickly code up some of the ideas here and validate the extensions are sufficient to enable them.

@cheery

This comment has been minimized.

Show comment
Hide comment
@cheery

cheery Mar 1, 2017

@cubanismo I schedule a study of these extensions for this weekend.

I'll see if I get texture streaming across two processes going.

Update: It seems I got both work and holiday events occupying my weekend, so I don't have full attention for this during at weekend. I'll also spend the next week by studying these extensions.

cheery commented Mar 1, 2017

@cubanismo I schedule a study of these extensions for this weekend.

I'll see if I get texture streaming across two processes going.

Update: It seems I got both work and holiday events occupying my weekend, so I don't have full attention for this during at weekend. I'll also spend the next week by studying these extensions.

@cheery

This comment has been minimized.

Show comment
Hide comment
@cheery

cheery Mar 7, 2017

@cubanismo Upgrading my NVIDIA drivers, but it appears the drivers date at 2017.2.14. They don't seem like recent enough to provide the extensions. I'm updating them anyway now, but it seems I won't be able to test. Update: VK_NV_external_memory is present. Update²: Oh it seems they are on the Windows drivers right now. I may be able to test if I update the drivers on different computer.

In any case I'm going to review the extensions, so don't worry about that.

Update³: OK. I needed the beta drivers for Linux, there they are.

import vulkan

print("instance extensions")
for name in vulkan.listInstanceExtensions()
    print("   ", name)

instance = vulkan.Instance({})

for dev in instance.listPhysicalDevices()
    print("device extensions")
    for name in dev.listExtensions()
        print("   ", name)

Output:

instance extensions
    VK_KHR_surface
    VK_KHR_xcb_surface
    VK_KHR_xlib_surface
    VK_KHR_display
    VK_EXT_debug_report
    VK_KHR_get_physical_device_properties2
    VK_KHX_device_group_creation
    VK_KHX_external_memory_capabilities
    VK_KHX_external_semaphore_capabilities
    VK_EXT_display_surface_counter
    VK_EXT_direct_mode_display
    VK_EXT_acquire_xlib_display
device extensions
    VK_KHR_swapchain
    VK_KHR_display_swapchain
    VK_KHR_descriptor_update_template
    VK_KHR_maintenance1
    VK_KHR_push_descriptor
    VK_KHR_shader_draw_parameters
    VK_KHR_sampler_mirror_clamp_to_edge
    VK_KHX_device_group
    VK_KHX_external_memory
    VK_KHX_external_memory_fd
    VK_KHX_external_semaphore
    VK_KHX_external_semaphore_fd
    VK_KHX_multiview
    VK_EXT_discard_rectangles
    VK_EXT_shader_subgroup_ballot
    VK_EXT_shader_subgroup_vote
    VK_EXT_display_control
    VK_NV_dedicated_allocation
    VK_NV_glsl_shader
    VK_NV_sample_mask_override_coverage
    VK_NV_viewport_array2
    VK_NV_viewport_swizzle
    VK_NV_geometry_shader_passthrough
    VK_NVX_device_generated_commands
    VK_NVX_multiview_per_view_attributes

All right, I got the bindings in place. I can start.

cheery commented Mar 7, 2017

@cubanismo Upgrading my NVIDIA drivers, but it appears the drivers date at 2017.2.14. They don't seem like recent enough to provide the extensions. I'm updating them anyway now, but it seems I won't be able to test. Update: VK_NV_external_memory is present. Update²: Oh it seems they are on the Windows drivers right now. I may be able to test if I update the drivers on different computer.

In any case I'm going to review the extensions, so don't worry about that.

Update³: OK. I needed the beta drivers for Linux, there they are.

import vulkan

print("instance extensions")
for name in vulkan.listInstanceExtensions()
    print("   ", name)

instance = vulkan.Instance({})

for dev in instance.listPhysicalDevices()
    print("device extensions")
    for name in dev.listExtensions()
        print("   ", name)

Output:

instance extensions
    VK_KHR_surface
    VK_KHR_xcb_surface
    VK_KHR_xlib_surface
    VK_KHR_display
    VK_EXT_debug_report
    VK_KHR_get_physical_device_properties2
    VK_KHX_device_group_creation
    VK_KHX_external_memory_capabilities
    VK_KHX_external_semaphore_capabilities
    VK_EXT_display_surface_counter
    VK_EXT_direct_mode_display
    VK_EXT_acquire_xlib_display
device extensions
    VK_KHR_swapchain
    VK_KHR_display_swapchain
    VK_KHR_descriptor_update_template
    VK_KHR_maintenance1
    VK_KHR_push_descriptor
    VK_KHR_shader_draw_parameters
    VK_KHR_sampler_mirror_clamp_to_edge
    VK_KHX_device_group
    VK_KHX_external_memory
    VK_KHX_external_memory_fd
    VK_KHX_external_semaphore
    VK_KHX_external_semaphore_fd
    VK_KHX_multiview
    VK_EXT_discard_rectangles
    VK_EXT_shader_subgroup_ballot
    VK_EXT_shader_subgroup_vote
    VK_EXT_display_control
    VK_NV_dedicated_allocation
    VK_NV_glsl_shader
    VK_NV_sample_mask_override_coverage
    VK_NV_viewport_array2
    VK_NV_viewport_swizzle
    VK_NV_geometry_shader_passthrough
    VK_NVX_device_generated_commands
    VK_NVX_multiview_per_view_attributes

All right, I got the bindings in place. I can start.

@cubanismo

This comment has been minimized.

Show comment
Hide comment
@cubanismo

cubanismo Mar 7, 2017

@cheery Thanks for taking a look. Be sure you're using the beta drivers from our Vulkan developer page:

https://developer.nvidia.com/vulkan-driver

You'll want the ones under the "Developer Beta Driver Downloads" section to get these features. They were last updated on February 28th at the time of this post.

cubanismo commented Mar 7, 2017

@cheery Thanks for taking a look. Be sure you're using the beta drivers from our Vulkan developer page:

https://developer.nvidia.com/vulkan-driver

You'll want the ones under the "Developer Beta Driver Downloads" section to get these features. They were last updated on February 28th at the time of this post.

@cheery

This comment has been minimized.

Show comment
Hide comment
@cheery

cheery Mar 7, 2017

@cubanismo I took some overview of what these functions do and how they should be used.

When creating a resource on the end that shares:

  • Capability queries before use.
  • Allocate external memory (VkExportMemoryAllocateInfoKHX)
  • Create image (VkExternalMemoryImageCreateInfoKHX)
  • Retrieve the FD or exported handle (vkGetMemoryFdKHX)

To retrieve a resource from shared handle

  • Retrieve all the associated information to use the resource.
  • vkGetMemoryFdPropertiesKHX
  • Import the memory using allocation function (VkImportMemoryFdInfoKHX)
  • Create image (VkExternalMemoryImageCreateInfoKHX), using the retrieved information.

It seems quite simple via the file descriptors. On win32 sharing the handle between any two processes doesn't seem as clear. Aside DuplicateHandle that requires me to know the handle of the target process, do I have a different way to pass the handle?

If I understood the spec right: The FD is consumed, but it must be closed explicitly if it is not successfully used. This is a bit weird requirement, how do I know inside one process that the other process consumed the handle?

cheery commented Mar 7, 2017

@cubanismo I took some overview of what these functions do and how they should be used.

When creating a resource on the end that shares:

  • Capability queries before use.
  • Allocate external memory (VkExportMemoryAllocateInfoKHX)
  • Create image (VkExternalMemoryImageCreateInfoKHX)
  • Retrieve the FD or exported handle (vkGetMemoryFdKHX)

To retrieve a resource from shared handle

  • Retrieve all the associated information to use the resource.
  • vkGetMemoryFdPropertiesKHX
  • Import the memory using allocation function (VkImportMemoryFdInfoKHX)
  • Create image (VkExternalMemoryImageCreateInfoKHX), using the retrieved information.

It seems quite simple via the file descriptors. On win32 sharing the handle between any two processes doesn't seem as clear. Aside DuplicateHandle that requires me to know the handle of the target process, do I have a different way to pass the handle?

If I understood the spec right: The FD is consumed, but it must be closed explicitly if it is not successfully used. This is a bit weird requirement, how do I know inside one process that the other process consumed the handle?

@ratchetfreak

This comment has been minimized.

Show comment
Hide comment
@ratchetfreak

ratchetfreak Mar 7, 2017

passing the FD can be done through a unix domain socket which will effectively dup() the fd just like DuplicateHandle will on win32.

Then the receiving process will know that the consuming succeeded through the return value.

Also in both cases the second handle/FD is owned by the receiving process and if it dies/crashes it will get closed automatically.

ratchetfreak commented Mar 7, 2017

passing the FD can be done through a unix domain socket which will effectively dup() the fd just like DuplicateHandle will on win32.

Then the receiving process will know that the consuming succeeded through the return value.

Also in both cases the second handle/FD is owned by the receiving process and if it dies/crashes it will get closed automatically.

@cubanismo

This comment has been minimized.

Show comment
Hide comment
@cubanismo

cubanismo Mar 9, 2017

Yes, on win32 you must somehow either give the receiving process the sending process's process handle, or give the sending process the receiving process's process handle, unless you're using the older "KMT"-style handles, which aren't recommended if the newer NT-style handles are available. I'm not a windows expert myself, so I can't recommend the best way of accomplishing that, but this is largely the same as any other type of win32 handle sharing, e.g., shared memory, file handles, etc., so I assume there are good methods out there.

cubanismo commented Mar 9, 2017

Yes, on win32 you must somehow either give the receiving process the sending process's process handle, or give the sending process the receiving process's process handle, unless you're using the older "KMT"-style handles, which aren't recommended if the newer NT-style handles are available. I'm not a windows expert myself, so I can't recommend the best way of accomplishing that, but this is largely the same as any other type of win32 handle sharing, e.g., shared memory, file handles, etc., so I assume there are good methods out there.

@cheery

This comment has been minimized.

Show comment
Hide comment
@cheery

cheery Mar 12, 2017

I made a blogpost for monday: Vulkan API is liberating the graphics from the desktop

The work so far has helped me to figure out how I should organize my Vulkan wrapper to make this use-case possible. So I haven't wasted any time by pursuing this subject. 👍

The language I'm working on is utilizing libuv for basic I/O, there's the pipes and uv_write2 which is said to support the HANDLE -scenario on win32 as well. I may be able to make an external_memory demo that is reusable and eventually portable across operating systems.

I didn't complete it this week, but I am motivated to keep looking into this. So far it seems doable and that the API would be workable. Here's the progress so far:

https://github.com/cheery/lever/blob/master/samples/warpgpu/external_memory_host.lc
https://github.com/cheery/lever/blob/master/samples/warpgpu/external_memory_client.lc

Next I'm going to allocate memory and a semaphore at the external_memory_client.lc and use a pipe to pass it to the external_memory_host.lc.

cheery commented Mar 12, 2017

I made a blogpost for monday: Vulkan API is liberating the graphics from the desktop

The work so far has helped me to figure out how I should organize my Vulkan wrapper to make this use-case possible. So I haven't wasted any time by pursuing this subject. 👍

The language I'm working on is utilizing libuv for basic I/O, there's the pipes and uv_write2 which is said to support the HANDLE -scenario on win32 as well. I may be able to make an external_memory demo that is reusable and eventually portable across operating systems.

I didn't complete it this week, but I am motivated to keep looking into this. So far it seems doable and that the API would be workable. Here's the progress so far:

https://github.com/cheery/lever/blob/master/samples/warpgpu/external_memory_host.lc
https://github.com/cheery/lever/blob/master/samples/warpgpu/external_memory_client.lc

Next I'm going to allocate memory and a semaphore at the external_memory_client.lc and use a pipe to pass it to the external_memory_host.lc.

@cheery

This comment has been minimized.

Show comment
Hide comment
@cheery

cheery Mar 14, 2017

Ok.. I appear to get a segmentation fault during attempting to allocate memory and I'm not sure what I'm doing wrong.

external_image_info = ffi.automem(vulkan.vk.ExternalMemoryImageCreateInfoKHX)
external_image_info[0] = {
    handleTypes = "OPAQUE_FD_BIT_KHX"
}
external_image = vulkan.Image(gpu, {
    pNext = external_image_info
    imageType = "2D"
    format = "R8G8B8A8_UNORM"
    extent = {width = 512, height = 512, depth = 1}
    mipLevels = 1
    arrayLayers = 1
    samples = "1_BIT"
    tiling = "OPTIMAL"
    usage = ["SAMPLED_BIT", "COLOR_ATTACHMENT_BIT"]
    initialLayout = "COLOR_ATTACHMENT_OPTIMAL"
})

req = external_image.getMemoryRequirements()
memoryIndex = null
for memtype in warpgpu.memory_report(gpu.physicalDevice).types
    if 1 & (req.memoryTypeBits >> memtype.index) != 0
        memoryIndex = memtype.index
        break
assert memoryIndex, "No memory available for this resource"

mem_info = ffi.automem(vulkan.vk.MemoryAllocateInfo)
export_info = ffi.automem(vulkan.vk.ExportMemoryAllocateInfoKHX)
export_info[0] = {
    handleTypes = "OPAQUE_FD_BIT_KHX"
}
mem_info[0] = {
    pNext = export_info
    allocationSize = req.size
    memoryTypeIndex = memoryIndex
}
mem = gpu.allocateMemory(mem_info)

Here's the inputs (on GTX960):

memoryIndex = 7
req.size = 1048576

I have the following extensions enabled:

"VK_KHR_get_physical_device_properties2",
"VK_KHX_external_memory_capabilities",
"VK_KHX_external_semaphore_capabilities",

"VK_KHX_external_memory",
"VK_KHX_external_memory_fd",
"VK_KHX_external_semaphore",
"VK_KHX_external_semaphore_fd",

All the function pointer handles are retrieved from the libvulkan.so, not via vkGetInstanceProcAddr or vkGetDeviceProcAddr, and it's from the 1.0.42.1 vulkan sdk.

The system fills sType -fields automatically, and I checked that the constants are correct. Am I missing something important, perhaps some query, or is this a bug?

cheery commented Mar 14, 2017

Ok.. I appear to get a segmentation fault during attempting to allocate memory and I'm not sure what I'm doing wrong.

external_image_info = ffi.automem(vulkan.vk.ExternalMemoryImageCreateInfoKHX)
external_image_info[0] = {
    handleTypes = "OPAQUE_FD_BIT_KHX"
}
external_image = vulkan.Image(gpu, {
    pNext = external_image_info
    imageType = "2D"
    format = "R8G8B8A8_UNORM"
    extent = {width = 512, height = 512, depth = 1}
    mipLevels = 1
    arrayLayers = 1
    samples = "1_BIT"
    tiling = "OPTIMAL"
    usage = ["SAMPLED_BIT", "COLOR_ATTACHMENT_BIT"]
    initialLayout = "COLOR_ATTACHMENT_OPTIMAL"
})

req = external_image.getMemoryRequirements()
memoryIndex = null
for memtype in warpgpu.memory_report(gpu.physicalDevice).types
    if 1 & (req.memoryTypeBits >> memtype.index) != 0
        memoryIndex = memtype.index
        break
assert memoryIndex, "No memory available for this resource"

mem_info = ffi.automem(vulkan.vk.MemoryAllocateInfo)
export_info = ffi.automem(vulkan.vk.ExportMemoryAllocateInfoKHX)
export_info[0] = {
    handleTypes = "OPAQUE_FD_BIT_KHX"
}
mem_info[0] = {
    pNext = export_info
    allocationSize = req.size
    memoryTypeIndex = memoryIndex
}
mem = gpu.allocateMemory(mem_info)

Here's the inputs (on GTX960):

memoryIndex = 7
req.size = 1048576

I have the following extensions enabled:

"VK_KHR_get_physical_device_properties2",
"VK_KHX_external_memory_capabilities",
"VK_KHX_external_semaphore_capabilities",

"VK_KHX_external_memory",
"VK_KHX_external_memory_fd",
"VK_KHX_external_semaphore",
"VK_KHX_external_semaphore_fd",

All the function pointer handles are retrieved from the libvulkan.so, not via vkGetInstanceProcAddr or vkGetDeviceProcAddr, and it's from the 1.0.42.1 vulkan sdk.

The system fills sType -fields automatically, and I checked that the constants are correct. Am I missing something important, perhaps some query, or is this a bug?

@cubanismo

This comment has been minimized.

Show comment
Hide comment
@cubanismo

cubanismo Mar 14, 2017

Look right from a glance. If you have easily compilable sample code, I can take a look locally.

cubanismo commented Mar 14, 2017

Look right from a glance. If you have easily compilable sample code, I can take a look locally.

@cheery

This comment has been minimized.

Show comment
Hide comment
@cheery

cheery Mar 14, 2017

I dropped it in there: https://github.com/cheery/lever

Quick instructions for compiling runtime on ubuntu and trying out the code above (takes 20min here):

python setup.py compile
./lever samples/warpgpu/external_memory_client.lc

API dump and GDB backtrace here: https://gist.github.com/cheery/05221ab3943e9b386e2f37a77b837b7a

cheery commented Mar 14, 2017

I dropped it in there: https://github.com/cheery/lever

Quick instructions for compiling runtime on ubuntu and trying out the code above (takes 20min here):

python setup.py compile
./lever samples/warpgpu/external_memory_client.lc

API dump and GDB backtrace here: https://gist.github.com/cheery/05221ab3943e9b386e2f37a77b837b7a

@cubanismo

This comment has been minimized.

Show comment
Hide comment
@cubanismo

cubanismo commented Mar 14, 2017

@cheery

This comment has been minimized.

Show comment
Hide comment
@cheery

cheery Mar 14, 2017

It was a forgotten .pyc file at here, combined with a stray dead import -statement. I fixed it. Run git pull origin master and try again.

I did git clean -xdf runtime/ and tried the compile to be sure, concluding there were not more old import statements that would haunt people (for now).

cheery commented Mar 14, 2017

It was a forgotten .pyc file at here, combined with a stray dead import -statement. I fixed it. Run git pull origin master and try again.

I did git clean -xdf runtime/ and tried the compile to be sure, concluding there were not more old import statements that would haunt people (for now).

@cheery

This comment has been minimized.

Show comment
Hide comment
@cheery

cheery Apr 7, 2017

Today I resumed looking up what's going on here. I'm suspecting that my program is crashing because I am doing a bad request that the card can't satisfy. But there is something else weird going on.

I've been tinkering around and trying out bunch of different functions now. vkGetPhysicalDeviceImageFormatProperties2KHR sometimes crashes if I give it only the ExternalImageFormatInfoKHX, but forget to give it the ExternalImageFormatPropertiesKHX into the return.

I tried to create an external semaphore. The moment I chain the additional structure in, it crashes.

Then I tried to add use PhysicalDeviceFeatures2KHR on device creation, chain it in too. Crashes too.

vk.getPhysicalDeviceProperties2KHR -> crash sometimes, sometimes not.

I've tried out several others, but just these ones seem to crash.

I updated to the NVIDIA beta driver 381.09 yesterday on my Linux desktop.

(Starting to search for some sample program in the web that's using the *device_properties2 extension, to see whether I can close away the language's vulkan bindings as a culprit here.)

Update: Wrote one program on my own. It does not manage to reproduce the issue. It seems to be isolated into my project now. Looks like I'll get it to work then. My guess is that it's an offset or size issue in structures.

cheery commented Apr 7, 2017

Today I resumed looking up what's going on here. I'm suspecting that my program is crashing because I am doing a bad request that the card can't satisfy. But there is something else weird going on.

I've been tinkering around and trying out bunch of different functions now. vkGetPhysicalDeviceImageFormatProperties2KHR sometimes crashes if I give it only the ExternalImageFormatInfoKHX, but forget to give it the ExternalImageFormatPropertiesKHX into the return.

I tried to create an external semaphore. The moment I chain the additional structure in, it crashes.

Then I tried to add use PhysicalDeviceFeatures2KHR on device creation, chain it in too. Crashes too.

vk.getPhysicalDeviceProperties2KHR -> crash sometimes, sometimes not.

I've tried out several others, but just these ones seem to crash.

I updated to the NVIDIA beta driver 381.09 yesterday on my Linux desktop.

(Starting to search for some sample program in the web that's using the *device_properties2 extension, to see whether I can close away the language's vulkan bindings as a culprit here.)

Update: Wrote one program on my own. It does not manage to reproduce the issue. It seems to be isolated into my project now. Looks like I'll get it to work then. My guess is that it's an offset or size issue in structures.

@cheery

This comment has been minimized.

Show comment
Hide comment
@cheery

cheery Apr 7, 2017

Awesome. I found out what it was.

The problem was records that were allocated but not cleaned when I set from a json dictionary without pool backing the allocation up. I changed this behavior in the language because it's not really the most insightful thing to not clean the struct fields when you SET something that is supposed to convert into a struct.

Since the fault was mine, I will proceed and write a full demo that is blitting an image across process boundaries.

cheery commented Apr 7, 2017

Awesome. I found out what it was.

The problem was records that were allocated but not cleaned when I set from a json dictionary without pool backing the allocation up. I changed this behavior in the language because it's not really the most insightful thing to not clean the struct fields when you SET something that is supposed to convert into a struct.

Since the fault was mine, I will proceed and write a full demo that is blitting an image across process boundaries.

@cheery

This comment has been minimized.

Show comment
Hide comment
@cheery

cheery Apr 9, 2017

I made it!

screenshot

The source code is in the samples/KHX_external_memory directory.

It's not entirely full because it doesn't run on Windows, doesn't do the proper round trips around the processes (the client must have to connect to recognize which physical device it starts, there is not that dependency flow now present), it's also not synchronizing with semaphores. I am surprised it works for that reason.

I can continue this sometime, but now I got to concentrate on something else. Overall I think this is quite easy to do compared to other Vulkan things. Also the property enumeration is all right and in obvious place. 💯 👌

cheery commented Apr 9, 2017

I made it!

screenshot

The source code is in the samples/KHX_external_memory directory.

It's not entirely full because it doesn't run on Windows, doesn't do the proper round trips around the processes (the client must have to connect to recognize which physical device it starts, there is not that dependency flow now present), it's also not synchronizing with semaphores. I am surprised it works for that reason.

I can continue this sometime, but now I got to concentrate on something else. Overall I think this is quite easy to do compared to other Vulkan things. Also the property enumeration is all right and in obvious place. 💯 👌

@cubanismo

This comment has been minimized.

Show comment
Hide comment
@cubanismo

cubanismo Apr 2, 2018

Glad to hear you got this working @cheery. I'm going to close this issue since the extensions required are no longer KHX/experimental and are included in Vulkan 1.1. Please re-open or file a new issue if there is anything else needed for this on the Vulkan side.

cubanismo commented Apr 2, 2018

Glad to hear you got this working @cheery. I'm going to close this issue since the extensions required are no longer KHX/experimental and are included in Vulkan 1.1. Please re-open or file a new issue if there is anything else needed for this on the Vulkan side.

@cubanismo cubanismo closed this Apr 2, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment