Feedback on Vulkan Portability Initiative

[Khronoswebmaster]

On February 26th, 2018, Khronos announced open source tools to bring Vulkan applications to Apple platforms as part of the Vulkan Portability Initiative. These tools include the free MoltenVK run-time library from The Brenwill Workshop, and a macOS port of the LunarG Vulkan SDK.

The Khronos Portability Initiative is defining and evolving a subset of Vulkan 1.0 that can be mapped at native performance levels to Metal and DX12. The released Apple tools are the first deliverables from this work.

Khronos and the members of the Vulkan Working Group have placed these tools into open source, and we welcome any feedback here on this issue or on the tool repositories below. We are particularly interested in functionality or performance issues that developers uncover, so we can work with the community to continually improve these porting solutions.

• MoltenVK Vulkan run-time for macOS and iOS
• LunarG Vulkan SDK for macOS

[procedural]

Please don't implement the portability layers in Rust, that will be a disaster!

[procedural]

Please don't implement the portability layers in Rust, that will be a disaster!

[msiglreith]

@procedural Out of interest, why not?

[msiglreith]

@procedural Out of interest, why not?

[rootext]

Is there public list of pitfalls/drawback Vulkan over MoltenVK in terms of converting Vulkan calls to Metal calls?
It would be great to have pages similar to
https://github.com/Microsoft/angle/wiki/Getting-Good-Performance-From-ANGLE and
https://github.com/Microsoft/angle/wiki/Known-Issues.

[rootext]

Is there public list of pitfalls/drawback Vulkan over MoltenVK in terms of converting Vulkan calls to Metal calls?
It would be great to have pages similar to
https://github.com/Microsoft/angle/wiki/Getting-Good-Performance-From-ANGLE and
https://github.com/Microsoft/angle/wiki/Known-Issues.

[kvark]

@procedural this is an unreasonable request. Anyone is free to implement Vulkan Portability initiative, it's is much welcome by the portability technical subgroup. You may choose Node.JS if that gets you desired performance and maintainability characteristics.

We believe that a portability layer has to be zero-overhead and robust against hitting undefined behavior, thread safety issues, and memory violation crashes. Rust can do this.

[kvark]

@procedural this is an unreasonable request. Anyone is free to implement Vulkan Portability initiative, it's is much welcome by the portability technical subgroup. You may choose Node.JS if that gets you desired performance and maintainability characteristics.

We believe that a portability layer has to be zero-overhead and robust against hitting undefined behavior, thread safety issues, and memory violation crashes. Rust can do this.

[hrydgard]

@rootext There's such a list of limitations here: https://github.com/KhronosGroup/MoltenVK/blob/master/Docs/MoltenVK_Runtime_UserGuide.md#known-moltenvk-limitations

[hrydgard]

@rootext There's such a list of limitations here: https://github.com/KhronosGroup/MoltenVK/blob/master/Docs/MoltenVK_Runtime_UserGuide.md#known-moltenvk-limitations

[kvark]

Khronos Group usually drops reference implementations for proposed initiatives

Consider this case to be unusual then. Portability sub-group was operating within the context of multiple implementations from the start. MoltenVK is not a Khronos reference implementation, it's an implementation of Metal side of portability subset.

What will be not hilarious is that people like me would have to reimplement it in C to integrate with already-written-in-C pipelines without injecting a non-ABI-stable compiler as yet another wacky dependency.

Do C/C++ have a stable ABI? Unstable Rust ABI would only be a concern if a Khronos-delivered library had Rust API. This is not the case here. gfx-portability compiles to a regular static/dynamic library that is not distinguishable from other libraries on the target system. No need to rewrite anything in C to use it.

Shove your thread safety you know where, threading is user's problem, not yours.

Well, this is just rude.
As you may know, some of the state in Vulkan driver/layer is internally synchronized. Getting this wrong leads to hard to reproduce issues, ranging from visual corruption to crashes and UB. A brief search over the young MoltenVK repo shows a few examples.

[kvark]

Khronos Group usually drops reference implementations for proposed initiatives

Consider this case to be unusual then. Portability sub-group was operating within the context of multiple implementations from the start. MoltenVK is not a Khronos reference implementation, it's an implementation of Metal side of portability subset.

What will be not hilarious is that people like me would have to reimplement it in C to integrate with already-written-in-C pipelines without injecting a non-ABI-stable compiler as yet another wacky dependency.

Do C/C++ have a stable ABI? Unstable Rust ABI would only be a concern if a Khronos-delivered library had Rust API. This is not the case here. gfx-portability compiles to a regular static/dynamic library that is not distinguishable from other libraries on the target system. No need to rewrite anything in C to use it.

Shove your thread safety you know where, threading is user's problem, not yours.

Well, this is just rude.
As you may know, some of the state in Vulkan driver/layer is internally synchronized. Getting this wrong leads to hard to reproduce issues, ranging from visual corruption to crashes and UB. A brief search over the young MoltenVK repo shows a few examples.

[dokipen3d]

Are compute shaders supported? (in the limitations it mentions geometry and tessellation are not supported so hopefully that means compute is).

[dokipen3d]

Are compute shaders supported? (in the limitations it mentions geometry and tessellation are not supported so hopefully that means compute is).

[nsubtil]

Wanted to remind everyone that this issue tracker is managed by the Khronos Group. Interactions here should follow the Khronos Code of Conduct (https://www.khronos.org/news/events/code-of-conduct), which prohibits aggressive or derogatory language. Please keep the discussion friendly and civil.

[nsubtil]

Wanted to remind everyone that this issue tracker is managed by the Khronos Group. Interactions here should follow the Khronos Code of Conduct (https://www.khronos.org/news/events/code-of-conduct), which prohibits aggressive or derogatory language. Please keep the discussion friendly and civil.

[BenFields724]

Got Vulkan up and running with GLFW on my mac machine without any problems. Nice to have some more support on the mac side of things! Nice Work!

[BenFields724]

Got Vulkan up and running with GLFW on my mac machine without any problems. Nice to have some more support on the mac side of things! Nice Work!

[jwtowner]

A portability layer in Rust would be difficult to use on platforms where we're restricted to using only the C/C++ toolchains provided with the platform devkits, or because it would require stakeholder buy-in on using Rust, which could be difficult to achieve. Think game consoles, UWP/Windows Store, Android.

When you're already running a C/C++ based shop, having to tool-up with Rust in order to build a portability layer would be more costly than if it were simply written in C/C++. It's another dependency in your build pipeline that can cause breakage, you need to train developers, etc.

With a little work, you can get Angle working for Xbox One, as it already supports UWP out-of-the box. A portability layer written exclusively in Rust just wouldn't work here, you'd be leaving a large segment of the industry out in the cold and having to come up with their own portability solution.

MoltenVK is great. Would love to see something that does the same for DX12.

[jwtowner]

A portability layer in Rust would be difficult to use on platforms where we're restricted to using only the C/C++ toolchains provided with the platform devkits, or because it would require stakeholder buy-in on using Rust, which could be difficult to achieve. Think game consoles, UWP/Windows Store, Android.

When you're already running a C/C++ based shop, having to tool-up with Rust in order to build a portability layer would be more costly than if it were simply written in C/C++. It's another dependency in your build pipeline that can cause breakage, you need to train developers, etc.

With a little work, you can get Angle working for Xbox One, as it already supports UWP out-of-the box. A portability layer written exclusively in Rust just wouldn't work here, you'd be leaving a large segment of the industry out in the cold and having to come up with their own portability solution.

MoltenVK is great. Would love to see something that does the same for DX12.

[kvark]

@jwtowner

A portability layer in Rust would be difficult to use on platforms where we're restricted to using only the C/C++ toolchains provided with the platform devkits, or because it would require stakeholder buy-in on using Rust, which could be difficult to achieve. Think game consoles, UWP/Windows Store, Android.

Which stakeholders do you mean? I believe all the nuts and bolts to deliver Rust code on those platforms are already in place, and it's a matter of making them polished and more accessible. Here is a quote from Rust Chucklefish whitepaper:

Even after taking into account the ten days needed to customize Rust for the Xbox, PS4, and Nintendo Switch consoles, Chucklefish saved time previously spent debugging cross-platform idiosyncrasies.

---

With a little work, you can get Angle working for Xbox One, as it already supports UWP out-of-the box. A portability layer written exclusively in Rust just wouldn't work here, you'd be leaving a large segment of the industry out in the cold and having to come up with their own portability solution.

Why wouldn't Rust code work there? UWP also doesn't look like a blocker: it has COM interface, and the bindings are already there.

In other words - yes, if your C++ shop wants to build and maintain a fork of the library, some Rust knowledge would (obviously) be helpful. But I don't see showstoppers to use one.

---

MoltenVK is great. Would love to see something that does the same for DX12.

FYI, there are a few D3D12 portability projects written in C++ in early stages: Rostkatze and VulkanOnD3D12, if you are so desperate :)

[kvark]

@jwtowner

A portability layer in Rust would be difficult to use on platforms where we're restricted to using only the C/C++ toolchains provided with the platform devkits, or because it would require stakeholder buy-in on using Rust, which could be difficult to achieve. Think game consoles, UWP/Windows Store, Android.

Which stakeholders do you mean? I believe all the nuts and bolts to deliver Rust code on those platforms are already in place, and it's a matter of making them polished and more accessible. Here is a quote from Rust Chucklefish whitepaper:

Even after taking into account the ten days needed to customize Rust for the Xbox, PS4, and Nintendo Switch consoles, Chucklefish saved time previously spent debugging cross-platform idiosyncrasies.

---

With a little work, you can get Angle working for Xbox One, as it already supports UWP out-of-the box. A portability layer written exclusively in Rust just wouldn't work here, you'd be leaving a large segment of the industry out in the cold and having to come up with their own portability solution.

Why wouldn't Rust code work there? UWP also doesn't look like a blocker: it has COM interface, and the bindings are already there.

In other words - yes, if your C++ shop wants to build and maintain a fork of the library, some Rust knowledge would (obviously) be helpful. But I don't see showstoppers to use one.

---

MoltenVK is great. Would love to see something that does the same for DX12.

FYI, there are a few D3D12 portability projects written in C++ in early stages: Rostkatze and VulkanOnD3D12, if you are so desperate :)

[jwtowner]

@kvark
Stakeholders such as publishers, funders, non-technical executives, and so on who have a stake in the intellectual property being developed. The more people you have to convince, especially in a studio where C++ is already entrenched, the more difficult such a task becomes and it can be very frustrating getting anywhere.

Sometimes, the reasons aren't technical ones, but rather due to business. Licensing agreements with publishers stopping you from deviating away from the straight and narrow provided by the devkits. Indie developers have a lot more freedom.

Rostkatze looks promising, thanks for the links.

[jwtowner]

@kvark
Stakeholders such as publishers, funders, non-technical executives, and so on who have a stake in the intellectual property being developed. The more people you have to convince, especially in a studio where C++ is already entrenched, the more difficult such a task becomes and it can be very frustrating getting anywhere.

Sometimes, the reasons aren't technical ones, but rather due to business. Licensing agreements with publishers stopping you from deviating away from the straight and narrow provided by the devkits. Indie developers have a lot more freedom.

Rostkatze looks promising, thanks for the links.

[Swyter]

It's pretty easy, if you want to make this thing portable you need to use what most game developers are using and not the other way around. That means plain and portable C/C++ with custom malloc(), properly modularized and with the bare minimum dependencies that make it work (POSIX and pthreads).

Otherwise you are only targeting hobbyists, armchair experts in the Hacker News bubble, and indies. Adding Rust libraries or any kind of exotic or immature integration that depends on specific compilers like Clang, even if they have ABI compatibility or a C API, is a no-go.

I understand that Mozilla is trying to push their language as much as possible. But that won't fly for most serious devs. They will just keep using their custom platform back-ends (DirectX/Metal/NVN/GNM/...) and nothing will change.

Take a page from SDL2.

[Swyter]

It's pretty easy, if you want to make this thing portable you need to use what most game developers are using and not the other way around. That means plain and portable C/C++ with custom malloc(), properly modularized and with the bare minimum dependencies that make it work (POSIX and pthreads).

Otherwise you are only targeting hobbyists, armchair experts in the Hacker News bubble, and indies. Adding Rust libraries or any kind of exotic or immature integration that depends on specific compilers like Clang, even if they have ABI compatibility or a C API, is a no-go.

I understand that Mozilla is trying to push their language as much as possible. But that won't fly for most serious devs. They will just keep using their custom platform back-ends (DirectX/Metal/NVN/GNM/...) and nothing will change.

Take a page from SDL2.

[kvark]

@Swyter I think you are mostly correct, but I have to fix a few points:

I understand that Mozilla is trying to push their language as much as possible.

No, you don't understand. Unlike MoltenVK, gfx-rs is a community project, and Mozilla just gives us a chance to be a part of the Khronos effort. There is no "push their language" here, other than some folks discriminating Rust on the basis of not being C/C++.

They will just keep using their custom platform back-ends (DirectX/Metal/NVN/GNM/...) and nothing will change.

Contrary to the rest of your comment, this is applicable to Vulkan Portability in general. And yes, I agree that it's most useful to indie and medium-sized teams. The most demanding developers already have their backends and can tune them better for their particular use cases.

Otherwise you are only targeting hobbyists, armchair experts in the Hacker News bubble, and indies.

A change needs to start somewhere. Fortunately, the industry is not set in stone. We see Ready At Down moving to Rust, EA's SEED looking into it, and more to come.

[kvark]

@Swyter I think you are mostly correct, but I have to fix a few points:

I understand that Mozilla is trying to push their language as much as possible.

No, you don't understand. Unlike MoltenVK, gfx-rs is a community project, and Mozilla just gives us a chance to be a part of the Khronos effort. There is no "push their language" here, other than some folks discriminating Rust on the basis of not being C/C++.

They will just keep using their custom platform back-ends (DirectX/Metal/NVN/GNM/...) and nothing will change.

Contrary to the rest of your comment, this is applicable to Vulkan Portability in general. And yes, I agree that it's most useful to indie and medium-sized teams. The most demanding developers already have their backends and can tune them better for their particular use cases.

Otherwise you are only targeting hobbyists, armchair experts in the Hacker News bubble, and indies.

A change needs to start somewhere. Fortunately, the industry is not set in stone. We see Ready At Down moving to Rust, EA's SEED looking into it, and more to come.

[Swyter]

Good to see that the language is at least gaining a little bit of popularity, but the amount of Rust evangelism surrounding it makes it look a bit like a cargo cult and makes me weary of all the Mozilla hype.

You can't just not use Rust in 2018™. There are many trendy system languages that come and go, but so far C is tried-and-tested and meets the sweet spot in production software, specially in games where performance tricks trump safety. You can of course program low-level graphics wrappers in Go, Rust or D, but we all know that it will be a big roadblock that will limit adoption to some degree.

If a proper C-family fork (DXVK-style) arises it will also lead to a waste of duplicated effort. It's cool to use it for research and experimental purposes like in Servo, but if the underlying APIs are built in C from firmware to user-mode driver and the thing that sits on top of your wrapper is C++ I don't think it adds much other than bloat and a complicated toolchain. C has powered the modern world for 46 years and it's still evolving.

I would just pay someone to make a DirectVK and reap the benefits. Less is more, I think.

So yeah, we will see how this turns out in a few years. As you can see many of us don't think it's the right tool for the job.

[Swyter]

Good to see that the language is at least gaining a little bit of popularity, but the amount of Rust evangelism surrounding it makes it look a bit like a cargo cult and makes me weary of all the Mozilla hype.

You can't just not use Rust in 2018™. There are many trendy system languages that come and go, but so far C is tried-and-tested and meets the sweet spot in production software, specially in games where performance tricks trump safety. You can of course program low-level graphics wrappers in Go, Rust or D, but we all know that it will be a big roadblock that will limit adoption to some degree.

If a proper C-family fork (DXVK-style) arises it will also lead to a waste of duplicated effort. It's cool to use it for research and experimental purposes like in Servo, but if the underlying APIs are built in C from firmware to user-mode driver and the thing that sits on top of your wrapper is C++ I don't think it adds much other than bloat and a complicated toolchain. C has powered the modern world for 46 years and it's still evolving.

I would just pay someone to make a DirectVK and reap the benefits. Less is more, I think.

So yeah, we will see how this turns out in a few years. As you can see many of us don't think it's the right tool for the job.

[msiglreith]

If a proper C-family fork (DXVK-style) arises it will also lead to a waste of duplicated effort.

I mean we (gfx-rs community) won't write a new C++ library and throw the existing code out of the window as our userbase are mostly Rust users atm.
It's not like Rust is superior for implementing one specific backend -> Vulkan library, actually C++ would be better here I agree. Mostly due to less 'FFI noise' when using Rust in my opinion. (I haven't seen some significant performance differences between gfx-rs's dx12 and rostkatze for some demo scenes)

If someone decides to write a C++ implementation or build upon of the existing early projects floating around I hope they can make use of our findings in writing an implementation. We are still in a stage where we experiment with different approaches and facing limitations of backends which we try to address. For us it's simply easier and faster to do this kind of work in Rust.

[msiglreith]

If a proper C-family fork (DXVK-style) arises it will also lead to a waste of duplicated effort.

I mean we (gfx-rs community) won't write a new C++ library and throw the existing code out of the window as our userbase are mostly Rust users atm.
It's not like Rust is superior for implementing one specific backend -> Vulkan library, actually C++ would be better here I agree. Mostly due to less 'FFI noise' when using Rust in my opinion. (I haven't seen some significant performance differences between gfx-rs's dx12 and rostkatze for some demo scenes)

If someone decides to write a C++ implementation or build upon of the existing early projects floating around I hope they can make use of our findings in writing an implementation. We are still in a stage where we experiment with different approaches and facing limitations of backends which we try to address. For us it's simply easier and faster to do this kind of work in Rust.