Planning for Vulkan, DirectX 12, and Metal

[tomspilman]

We need to plan how we want to approach supporting the new graphics APIs: Vulkan, DirectX 12, and Metal.

These new APIs are more verbose and require more setup code to do similar things as previous graphics libraries. This makes MonoGame even more useful to developers if we support them well.

We should consider this work in two steps:

• Implement new graphics backends within our existing XNA style API.
• Design and implement a new API that fits these new graphics standards.

The most immediate thing would be shoehorning these into our existing XNA style APIs. There are potentially (I will know for sure soon) some performance benefits to be gained just by doing that. Likely this will also require support in the content pipeline for new shader formats as well.

The next bigger step is designing a new graphics API for MonoGame that better aligns with Vulkan/DX12/Metal features. It would be nice to support both this new API and our existing one at the same time, but we could just make a clean break if we feel confident about it.

[theZMan]

Here's a fun snippet that shows you all the code needed to render a single triangle using Vuican in windows

https://github.com/SaschaWillems/Vulkan/blob/master/triangle/triangle.cpp

I suspect DX12 is probably similar

He also has many other examples https://github.com/SaschaWillems/Vulkan - a useful resource

[tomspilman]

all the code needed to render a single triangle using Vuican

Yep... basically with all these APIs they have removed a lot of code the graphics driver did and left the work to the game developer. This is really where most of the performance from these APIs comes from.

[lithiumtoast]

Apple released documentation and examples for Metal here.
Xamarin has also examples for Metal in C# with their API wrapper. See here and here.

[lithiumtoast]

Metal is available for iOS devices using the Apple A7 or later (iPhone 5s or later), as well as Macs (2012 models or later) running OS X El Capitan.

[tomspilman]

@lithiumtoast - Thanks for that... lets start a new issue in particular for the first bit of Metal support. #4593

@KonajuGames @dellis1972 - Do you think it is worth figuring out our new post-XNA graphics API first before we try to implement Metal/Vulkan/DX12 under the existing XNA API?

[lithiumtoast]

@tomspilman What about Mac support for Metal? All Macs that are 2012 or newer are Metal supported by Apple as long as they running OS X El Capitan. Could we branch off of the MacOS project to support Metal since OpenGL is having issues anyways? (see #3581)

[tomspilman]

What about Mac support for Metal?

Well we just in the last few months have moved to unify the Linux/Mac/Windows GL builds into a single binary that works on all platforms. Moving to Metal on OSX would mean loosing that unification and adding back another platform.

I think for now we wait to see if Vulkan makes it to Mac. Apple hasn't commented on this yet... so it could go either way I think.

[lithiumtoast]

I think for now we wait to see if Vulkan makes it to Mac.

@tomspilman Fair enough, though it kinda sucks cause if we are going to do Metal for iOS the code would be identical for Mac.

[bonesoul]

also this seems interesting; https://github.com/jwollen/SharpVulkan

[KonajuGames]

also this seems interesting; https://github.com/jwollen/SharpVulkan

Which won't get confused with https://github.com/mono/VulkanSharp at all. :)

[KonajuGames]

It will be the same story with Vulkan on PC as it is for Metal on iOS and Mac. Not all PCs will support Vulkan. My PC has a GeForce 500 series card, which is still perfectly fine for every game I want to play, but Nvidia provide no Vulkan support for it. It will also be the same on Android. It requires a certain GPU to get Vulkan support. I think in both Vulkan and Metal support we need to provide an OpenGL fallback or we alienate a large part of the market. Even waiting until the end of 2016 won't be enough.

[KonajuGames]

Though I did just find that LunarG have released Vulkan drivers for a wider range of Windows and Linux (for Ubuntu 14+ flavoured versions of Linux).

[NemoKradXNA]

Hi all, very interested in this, I have some experience with XNA, but for the past few years have been playing with Unity3D, have also got into compute and geom shaders using Unity, but I find I am missing writing my own engine and the flexibility (as well as some of the headaches) that come with that.

So, I am very interested to see how this pans out, I have a notion of creating an engine based off MonoGame giving it a similar Editor like Unity has, but without all the Unity Idiosyncrasies, and would really not like to be locked into DX9 :)

Not sue I can give a great deal of input at the moment, as I need to get my head back into XNA/MonoGame, but thought I would say hi and express my interest :D

Looking forward to see what happens with this, and hope I can give some constructive input later on :)

Charles.

[bonesoul]

any updates on this?

[tomspilman]

any updates on this?

Not really... we still want to do it, but need to work out a plan first.

Ideally we would have a few different design proposals from contributors as to what this new API should look like. These proposals should be the basic API skeleton as a repo, a Gist, or a Google Doc.

We can then discuss the proposals and design the final API as a repo/gist/doc.

Only once the final API is agreed on will we even consider any code submissions on this issue. If anyone submits a PR with code at this time we will just reject it. We won't design an API as we go here... we need a well considered design before any work begins.

[Jjagg]

Do you still want to support Vulkan/Metal backends with the existing API first @tomspilman? I'm interested in contributing in both that and designing the new API. There are no Vulkan drivers for my GPU though :( And I have an iPhone 4 (I know...) and no Mac so I can't test Metal either.

On a seperate note: #4720 might be a good start for supporting other shading languages.

[tomspilman]

you still want to support Vulkan/Metal backends with the existing API first @tomspilman?

Sure... if someone contributed a new backends for the existing API we would likely accept it. Specially if it shows to perform as well if not better than the existing implementations for OpenGL or DirectX.

#4720 might be a good start for supporting other shading languages.

Not really.... new language support is mainly just a matter of extending the 2MGFX tool like I have for PS4, PSVita, and Xbox One shading languages already. If I could ever find some time I would document how to do this.

[KonajuGames]

There are no Vulkan drivers for my GPU though

Have you looked at the LunarG Vulkan SDK for Windows and Linux? It's not restricted to certain GPUs like NVidia's drivers.

[KonajuGames]

And I see I linked to LunarG a while back.

[Jjagg]

Have you looked at the LunarG Vulkan SDK for Windows and Linux?

Oh, I've seen that page before but haven't actually tried this. I'll check this out again, thanks!

[nkast]

I'd like to comment on the initial post of #2772 and propose a slightly different approach.

First, how such a change will look in code. (public API)

ex.1

this.GraphicsDevice.BeginDeferredContent();
// draw Models, Sprites or Vertices.
_model.Draw(_world, _view, _projection);
var cmdList = this.GraphicsDevice.EndDeferredContent();
GraphicsDevice.Execute(cmdList);

This will draw to a deferred content and then execute it on the immediate context (or merge it with the current deferred context if we were to allow nested contexts). The above will work on both the main thread and on working threads.

ex.2

var t1result = GraphicsDevice.DrawAsync(() => { test1.Draw(gameTime); });
var t2result = GraphicsDevice.DrawAsync(() => { test2.Draw(gameTime); });
var t3result = GraphicsDevice.DrawAsync(() => { test3.Draw(gameTime); });
var t4result = GraphicsDevice.DrawAsync(() => { _model.Draw(_world, _view, _projection); });
// do other drawing on the main thread/context
GraphicsDevice.Execute(t1result);
GraphicsDevice.Execute(t2result);
GraphicsDevice.Execute(t3result);
GraphicsDevice.Execute(t4result);

GraphicsDevice.DrawAsync() will run test1.Draw() on a separate thread/context and return a Task. It could accept an action, an IDrawable or a delegate with signature DrawDelegate(GameTime).
GraphicsDevice.Execute() awaits for t1result and execute it on the immediate context.

The idea is to let GraphicsDevice provide the current context on a per thread basis.
This can be done by changing _d3dContext into [ThreadStatic] private static DeviceContext _d3dContext;. We must also wrap DeviceContext along with Dirty flags, Vertex/Index buffers into a separate GraphicsContext Class.
The main difference from #2772 is that GraphicsDevice will not inherit from GraphicsContext and we don't need to expose GraphicsContext and its methods. GraphicsDevice has to forward all context related updates to the current content through getters/setters.

Plus

1. No need to modify drawing code. It's easy to convert existing code by wrapping it into a DrawAsync() action or BeginDeferred()/EndDeferred(). GraphicsDevice is responsible to use the right context.
   On MG codebase we don't need new methods likeModel.Draw(GraphicsContext, ,,,), Effectpass.Apply(GraphicsContext), Effect.Apply(GraphicsContext), etc.
2. Thread isolation of _context.
   We can remove all lock(_context) since it's impossible to access _context from other threads.
3. We can write extensions for XNA and MG backward compatibility that will work with the new code (action/task will be executed synchronously from .Execute(task<GraphicsContext>)).

Minus

1. It doesn't support multiple GraphicsDevice instances. Since _d3dContext is now static, is shared among all instances. There's an easy workaround, we can store the immediateContext in a member field and the currentContext on a ``[ThreadStatic]` field.

internal DeviceContext CurrentContext 
{
   get { return _currentContext ?? immediateContext;}
}

We probably need to implement a stack anyway to support nested context. With this change we no longer have thread isolation, but we can enforce it with a couple of checks and remove the locks anyway. Working threads will have to draw on a deferred context and dispatch command list and drawing to the main thread.

I have an early test to demostrate this. It works with the current DX11 but the API can be extended to DX12/Vulcan.

[KonajuGames]

Nice work. You've put some thought into this, and made a working example.

[nkast]

The test is not 100% correct. A lot of hacks to baypass dirty flags and exceptions.
First thing would be to refactor GraphicsDevice and implement GraphicsContext.

[mrhelmut]

There are a bunch of options around, like Veldrid and such. Everyone likely has their own favorite but...

There are a few factors that we will be weighting in before proceeding:

• we'd like to avoid depending on a third-party to maintain that native layer, we had our share of issues with that in the past and doing our own fork is not viable either because the delta might prevent us from pulling upstream improvements;
• we'd like the native layer to be minimal, with only the bare-minimum code for MonoGame/FNA to work, so that maintainability is smooth and we can fast iterate on it;
• it should be easy to add new APIs whenever something new comes out, like game consoles (and porting something like Veldrid or WebGPU to consoles sounds like a choir compared to a minimal layer)

As of now, what's making the most sense, is a common involvement in FNA3D.

[ghost]

Those are good points, the WebGPU specification doesn't have NVN or GNM listed as backends, for example.

If FNA3D were to be used, would that not mean completely scrapping MGFX? Correct me if I'm wrong, but MGFX transpiles HLSL to DXIL and GLSL on build, whereas FNA3D directly uses DXIL and transpiles to GLSL at runtime. Not to mention MonoGame also adds some metadata to the output like the MonoGame assembly name, which is not compatible since FNA3D is not expecting any metadata when parsing.

On the bright side, integrating FNA3D should allow MonoGame to remove some preprocessor directives that have long since overstayed their welcome 🥰

[mrhelmut]

If FNA3D were to be used, would that not mean completely scrapping MGFX? Correct me if I'm wrong, but MGFX transpiles HLSL to DXIL and GLSL on build, whereas FNA3D directly uses DXIL and transpiles to GLSL at runtime. Not to mention MonoGame also adds some metadata to the output like the MonoGame assembly name, which is not compatible since FNA3D is not expecting any metadata when parsing.

That is indeed the fondation of the difference between FNA and MonoGame, but yeah, MonoGame wishes to keep the content pipeline and 2MGFX so we'll have to figure out how to make FNA3D to consume the shader binaries.

remove some preprocessor directives that have long since overstayed their welcome 🥰

Yes! That is the whole purpose of this project: to remove platform code and having the C# API be a single common high level library.

[flibitijibibo]

One possible path for MGFX when targeting FNA3D is to make it similar to FXC - this is one thing we've actually had some difficulty with, because we technically can't distribute fxc.exe from the DXSDK, but at the same time fxc just calls D3DCompile and ID3DXEffectCompiler and not much else, so MGFX being an alternative comipler frontend would still be really useful. As long as FNA has D3D9-era DXBC and roughly follows the Effects Framework binary format, we won't know the difference.

Another path is to skip FNA3D altogether and go straight to SDL_gpu. We won't necessarily be doing this since we really like the ABI we made, but it's something we're heavily invested in right now:

https://github.com/FNA-XNA/FNA/wiki/Appendix-F:-Upcoming-Support-Changes#fna3d-and-sdl_gpu

[ghost]

@mrhelmut I think Ethan's solution is the most likely. The way FNA3D handles effects with Mojoshader is incompatible with how MG handles effects (asides from DXIL output with the Windows configuration most likely).

However, this would also limit MG to D3D9-era DXBC, which might be quite literally the opposite of modernizing 2MGFX

[mrhelmut]

Yeah, there is going to be some choices to be made in regard to where we'd like to take MonoGame. For instance, working with FNA3D likely means that we don't expand the graphics pipeline of MG (e.g. cpt-max' work wouldn't make it), and expanding the graphics pipeline means we have an enormous work to do with consoles (which is why max' work is taking so long to be reviewed).

In other news, we have both the Vulkan and D3D12/GDK backends in the work, for the time being independently of merging the backend into a single native library.

[ghost]

A MG3D native layer for Vulkan, D3D12, and Metal is something I am personally interested in working on. It could inherit a lot of FNA3D architecture but with ShaderConductor (most likely) instead of MojoShader. I personally would like to see MG move away from XNA, which I think is common sentiment anyways #3306

Is this something you would be interested in reviewing if I got a MVP working?

[mrhelmut]

We kind of need to keep an XNA-compatible version around because there are still surprisingly (big) games using it, but yes, MonoGame should depart from it at some point. My though would be to set that for 3.9 or 4.0, call that the "legacy LTS branch", and jump to 5.0 which would start breaking stuff.

But breaking stuff is independent of working on that "MG3D" or the Vulkan/D3D12 drivers.

[mrhelmut]

Vulkan and D3D12 are indeed APIs with which you can stall stuff easily if you don't know what you're doing. That person talks about pipeline caching (not hardware diversity), which is the basics of those APIs. If you indeed re-create the pipeline each time a pipeline state change, you are going to have a very bad time with performances (Elden Ring is notorious for its pipeline stuttering and runs better with Steam Deck emulation because it caches pipelines and effects). Pipelines are designed to be created once, and cached as much as possible. That we are aware of, and the plan is not to half-do things, if that's the concern. And I don't think that frame pacing will be an issue with a Vulkan/D3D12 MonoGame because I don't expect that we will have to go into too complex swap chains, we mostly need a basic swap chain with one command queue, so if we manage in-flight frames of active waiting the sync, things should be smooth.

[mrhelmut]

Silk is a binding, not an implementation, it wouldn't help in this case. And it doesn't cover consoles (which can't quite work with bindings, we have to write our own native driver anyway).

[Beyley]

@damian-666
Silk maintainer here, we do not provide an abstraction like Monogame, and it's not in the plans to do so, we also do not provide a way for "unifying the graphics pipelines" and for shader transpilation you will unfortanately have to look elsewhere (Veldrid.SPIRV is a good one)
Consoles are also not possible right now, and likely won't be unless someone takes the initiative. The goals of Silk and Monogame are very different

also side note: our DX bindings already work, and so do our Windowing/Input APIs

[MrScautHD]

It will never release :(

[Beyley]

It will never release :(

it's a hard thing to get right with multiple possible paths forward, comments like this are unhelpful, if you want it, take the initiative and start working on it yourself

[afuzzyllama]

Is there any documentation started for potential paths forward?

…

On Sun, Nov 27, 2022, 1:37 PM Beyley Thomas ***@***.***> wrote: It will never release :( it's a hard thing to get right with multiple possible paths forward, comments like this are unhelpful, if you want it, take the initiative and start working on it yourself — Reply to this email directly, view it on GitHub <#4571 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGQV52ZKKJAAADOFFEUSPDWKOTABANCNFSM4B4QCQ3Q> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Beyley]

Is there any documentation started for potential paths forward?

On Sun, Nov 27, 2022, 1:37 PM Beyley Thomas @.***>
wrote:

It will never release :(

it's a hard thing to get right with multiple possible paths forward,
comments like this are unhelpful, if you want it, take the initiative and
start working on it yourself

—
Reply to this email directly, view it on GitHub
#4571 (comment),
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGQV52ZKKJAAADOFFEUSPDWKOTABANCNFSM4B4QCQ3Q
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

read this thread

[mrhelmut]

This is common to all things internet, but if you'd like to see things moving forward, being antagonistic never is the right move. Showing defiance only contributes to lower moral and ultimately seeing the people invested in a project to just leave.

This goes for open-source, video games, and other things. If you are waiting for something, show some enthusiasm instead.

Negativity only helps when it's based on a constructive feedback, otherwise it's just doing the opposite of what you'd be hoping for.

Vulkan and D3D12 are being worked on privately (no ETA). Metal is up for grab.

Please be aware that these are big projects. Big enough to require that we work on it only if there is a party willing to pay for it.

[Muhammadinaam]

Vulkan and D3D12 will be good addition. Contributes, please carry on good work. Thanks.

[mrhelmut]

Quick update on that:

• Metal is not a priority because Vulkan can target Metal with MoltenVK, which we plan to use (it's less maintenance to work on one target that fits many than trying to have them all)
• Vulkan is almost done (it is salvaged from a canceled private platform for which Tom did a Vulkan implem) but it need some work before it can be moved to the public repository
• D3D12 is the one that need a lot of work and the Foundation will issue a bounty in the coming days to make it move forward

[Muhammadinaam]

Quick update on that:

• Metal is not a priority because Vulkan can target Metal with MoltenVK, which we plan to use (it's less maintenance to work on one target that fits many than trying to have them all)
• Vulkan is almost done (it is salvaged from a canceled private platform for which Tom did a Vulkan implem) but it need some work before it can be moved to the public repository
• D3D12 is the one that need a lot of work and the Foundation will issue a bounty in the coming days to make it move forward

I would like to contribute to the development fund for Vulkan and D3D12. This is a great framework.

[mengusfungus]

Is there any preliminary information on what the next gen abstraction interface might look like? Specifically I'm interested to know what the shader model will look like, I'm currently assuming it will still be hlsl (which version?) and there will be under the hood transpiling to spir-v if it's a vulkan backend (or metal through moltenvk). Will it support dx12-like raytracing intrinsics at first or will that be another release further down the line?

[tomspilman]

Is there any preliminary information on what the next gen abstraction interface might look like?

@harry-cpp is working on this. I think there is at least one PR for it out there.

Any updates @harry-cpp ?

I'm currently assuming it will still be hlsl (which version?) and there will be under the hood transpiling to spir-v if it's a vulkan backend (or metal through moltenvk).

Yes... likely HLSL 4/5/6 will be supported for a few reasons:

1. It is easier to go from HLSL to glsl/Spir-V than the other way around.
2. XNA was always HLSL and we want to preserve that history.
3. HLSL is supported natively on more consoles than Spir-V.
4. HLSL is more established (been around more than 20 years).

The transpiling will still occur during content builds because shader programs are getting more and more complex and compile times there matter. Also because some platforms do not support compiling shaders during gameplay.

Will it support dx12-like raytracing intrinsics

Hard to say... not many other platforms have "raytracing" as a first class feature like that.

Ideally yes, but it would be something that is not portable across platforms which is one of our key goals.

[mengusfungus]

My understanding is that vulkan raytracing extensions were specd to allow direct compilation from modern hlsl to spir-v and DXC implements this but admittedly I know nothing about the situation on mobile as that’s never been my focus. It would definitely be huge if achievable esp since there are already areas where MG users need to be mindful of underlying platform capabilities. Thanks for the response looking forward to updates.