Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API to accept and use precompiled shader #2861

Open
5 of 11 tasks
hajimehoshi opened this issue Dec 16, 2023 · 8 comments
Open
5 of 11 tasks

API to accept and use precompiled shader #2861

hajimehoshi opened this issue Dec 16, 2023 · 8 comments

Comments

@hajimehoshi
Copy link
Owner

hajimehoshi commented Dec 16, 2023

Operating System

  • Windows
  • macOS
  • Linux
  • FreeBSD
  • OpenBSD
  • Android
  • iOS
  • Nintendo Switch
  • PlayStation 5
  • Xbox
  • Web Browsers

What feature would you like to be added?

On consoles, compiling shaders is possible but takes long (Xbox), or, is probably impossible (PS5) 😢. We might need a tool to precompile shader and APIs to use them...

Why is this needed?

For some special environments where dynamic compiling shaders is difficult.

@f9x0
Copy link

f9x0 commented Dec 19, 2023

Hello. Consider adding an API to use your shaders.
Without a doubt, this feature will be for experienced developers and will open the way to the world of shaders. Naga, wgsl and other community tools. I like Kage, but it's good to have an alternative..

@hajimehoshi
Copy link
Owner Author

I'm afraid this suggestion is not for enabling other languages than Kage

@hajimehoshi
Copy link
Owner Author

hajimehoshi commented Dec 24, 2023

To minimize an impact of the existing APIs, I suggest this API:

// AddPrecompiledShader adds a pair of a shader source and a shader binary.
// When a new shader is created by NewShader,
// Ebitengine tries to seek the pair that has the same shader source.
// If there is such a pair, Ebitengine uses a registered shader binary for the same shader source instead of compiling it.
//
// AddPrecompiledShader is useful to shoten the shader compile time, or even necessary in an environment
// where you cannot compile a shader on the fly.
func AddPrecompiledShader(kageShaderSource []byte, nativeShaderBinary []byte, graphicsLibrary GraphicsLibrary) error

EDIT: Hmm, how can we treat vertex shader?

@hajimehoshi
Copy link
Owner Author

OpenGL: It is impossible to precompile shaders. AddPrecompiledShader should always return an error.

DirectX: A native shader binary is the result of D3DCompile. This function returns ID3DBlob, and this can be converted to a byte slice, and be created via D3DCreateBlob. A vertex shader and a pixel shader are separated.

Metal: A native shader binary is a .metallib format. This can be used at makeLibrary for NSData. See also https://developer.apple.com/documentation/metal/shader_libraries/building_a_shader_library_by_precompiling_source_files. A vertex shader and a fragment shader are NOT separated. So, Ebitengine would have to expose the binary for a vertex shader? Hmm.

@hajimehoshi
Copy link
Owner Author

Instead of exposing the the vertex shader, would this work?

func CompileShaderToNativeSource(kageShaderSource []byte, graphicsLibrary GraphicsLibrary) ([]byte, error)

// Compiling a native source to a native binary is a developer's responsibility.

func AddPrecompiledShader(kageShaderSource []byte, nativeShaderBinary []byte, graphicsLibrary GraphicsLibrary) error

@hajimehoshi hajimehoshi modified the milestones: v2.7.0, v2.8.0 Mar 12, 2024
@hajimehoshi
Copy link
Owner Author

hajimehoshi commented Apr 14, 2024

Compiling and using a shader

in a pseudo Go-like code.

OpenGL

vertex_shader_source, fragment_shader_source = glsl.Compile(ir)
vertex_shader = glNewShader(vertex_shader_source)
fragment_shader = glNewShader(fragment_shader_source)
program = glNewProgram(vertex_shader, fragmen_shader, array_buffer_layout_names)

Metal

src = metal.Compile(ir)
lib = MakeLibrary(src)
vertex_function = MakeFunction(lib, vertex_name)
fragment_function = MakeFunction(lib, fragment_name)

See also:

DirectX

vertex_shader_source, pixel_shader_source, offsets = hlsl.Compile(ir)
vertex_blob = D3DCompile(vertex_shader, name, ...)
pixel_blob = D3DCompile(pixel_shader, name, ...)

See also:

Proposal (TBD)

As there are various ways to compile and use shaders in different environments, the precompilation APIs should be separate for each environment unfortunately.

package shaderprecomp

func BuiltinShaderSources() [][]byte
 
func CompileToHLSL(kageSource []byte) (vertex []byte, pixel []byte, err error)

func RegisterDirectXShaderBlobs(kageSource []byte, vertexBlob []byte, pixelBlob []byte)

func CompileToMSL(kageSource []byte) ([]byte, error)

func RegisterMetalShaderLibrary(kageSource []byte, library []byte)

hajimehoshi added a commit that referenced this issue May 4, 2024
The current implementation is only for macOS so far.

Updates #2861
hajimehoshi added a commit that referenced this issue May 4, 2024
The current implementation is only for macOS so far.

Updates #2861
hajimehoshi added a commit that referenced this issue May 6, 2024
hajimehoshi added a commit that referenced this issue May 6, 2024
@hajimehoshi
Copy link
Owner Author

hajimehoshi commented May 6, 2024

The final API is like this

package shaderprecomp

type ShaderSource struct {
    // ...
}
func NewShaderSource([]byte) (*ShaderSource, error)
func (s *ShaderSource) ID() ShaderSourceID

type ShaderSourceID [16]byte
func (s *ShaderSourceID) String() string

func AppendBuiltinShaderSources(sources []*ShaderSource) []*ShaderSource
 
func CompileToHLSL(vertexWriter, pixelWriter io.Writer, kageSource *ShaderSource) error

func RegisterFXCs(id ShaderSourceID, vertexFXC []byte, pixelFXC []byte)

func CompileToMSL(w io.Writer, kageSource *ShaderSource) error

func RegisterMetalLibrary(id ShaderSourceID, library []byte)

hajimehoshi added a commit that referenced this issue May 26, 2024
This reverts commit ecc3f29.

Reason: we are considering to remove ShaderSourceID

Updates #2861
Updates #2999
hajimehoshi added a commit that referenced this issue May 26, 2024
This change simplifies the APIs to avoid some confusions around IDs.

Updates #2861
Closes #2999
hajimehoshi added a commit that referenced this issue May 26, 2024
`ShaderSourceID` was confusing as there was no guarantee the same ID is
used for the same source if Ebitengine versions are different.

`ShaderSource` should be kept as the built-in shader contents should not
be exposed.

Updates #2861
Closes #2999
hajimehoshi added a commit that referenced this issue May 26, 2024
`ShaderSourceID` was confusing as there was no guarantee the same ID is
used for the same source if Ebitengine versions are different.

`ShaderSource` should be kept as the built-in shader contents should not
be exposed.

Updates #2861
Closes #2999
@hajimehoshi hajimehoshi reopened this Aug 18, 2024
@hajimehoshi
Copy link
Owner Author

hajimehoshi commented Aug 18, 2024

With #2984, I realized some considerations:

  • Builtin shaders don't have to or should not be exposed.
  • The shader complation processes like what the examples/shaderprecomp does can be provided by this package, and CompileTo* might able to be hidden.

Thus, until I decide a better API design, I'd like to retract the package once. I might postpone the milestone for release.

hajimehoshi added a commit that referenced this issue Aug 18, 2024
With #2984, we realized some considerations:

* Builtin shaders don't have to or should not be exposed.
* The shader complation processes like what the examples/shaderprecomp
  does can be provided by this package

Thus, until we decide a better API design, we'd like to retract the
package once.

Updates #2861
@hajimehoshi hajimehoshi modified the milestones: v2.8.0, v2.9.0 Sep 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants