support more shader stages #263

ukari · 2021-02-17T09:59:51Z

9.5. Vertex Shaders
9.6. Tessellation Control Shaders
9.7. Tessellation Evaluation Shaders
9.8. Geometry Shaders
9.9. Fragment Shaders
9.10. Compute Shaders
9.11. Ray Generation Shaders
9.12. Intersection Shaders
9.13. Any-Hit Shaders
9.14. Closest Hit Shaders
9.15. Miss Shaders
9.16. Callable Shaders

vulkan mesh + task spec

expipiplus1 · 2021-02-18T02:01:05Z

Thanks!

Would it be possible for you to manually test a ray tracing shader? When I used these last I had to specify the SPIR-V version: https://github.com/expipiplus1/vulkan/blob/master/examples/rays/Pipeline.hs#L189

ukari · 2021-02-18T19:08:07Z

It seems some comments about compileShaderQ is wrong, so I fix it.

I test rgen.glsl shader in both shaderc/glslc and glslang/glslangValidator in shell, it needs spirv1.4. So I passed needed flag which could specify spirv1.4 for functions about ray tracing shaders. And now compileShader and compileShaderQ in ShaderQQ/Shaderc.hs have different signatures, I guess there should be a major version upgrade due to this api changed.

For task/mesh shaders, I test and found that they could be compile to spirv1.0, so no changes made on them.

Here is the test shaders. I have few knowledge about hlsl so I use a example from wihlidal blog, and my test on rgen.hlsl returns rgen.hlsl: rgen.hlsl(12): error at column 31, HLSL parsing failed.. Consider that the spec says that Vulkan 1.1 and SPIR-V 1.4 are needed for ray tracing, I didn't test hlsl anymore.

ukari · 2021-02-18T19:16:00Z

I noticed that both glslang/glslangValidator and shaderc/glslc could compile both glsl and hlsl.

What is the purpose for Utils/ShaderQQ.hs and Utils/ShaderQQ/Shaderc.hs use different glsl/hlsl shader to spirv compile tool?

Or should there be a refactor like the stuffs below?

- Vulkan/Utils/Shader/Common.hs
- Vulkan/Utils/Shader/GLSL/Shaderc.hs
- Vulkan/Utils/Shader/GLSL/Glslang.hs
- Vulkan/Utils/Shader/HLSL/Shaderc.hs
- Vulkan/Utils/Shader/HLSL/Glslang.hs

expipiplus1 · 2021-02-19T02:07:42Z

Thanks for all this!

Or should there be a refactor like the stuffs below?

- Vulkan/Utils/Shader/Common.hs
- Vulkan/Utils/Shader/GLSL/Shaderc.hs
- Vulkan/Utils/Shader/GLSL/Glslang.hs
- Vulkan/Utils/Shader/HLSL/Shaderc.hs
- Vulkan/Utils/Shader/HLSL/Glslang.hs

That sounds like an excellent idea! It is the way it is currently because I wanted an example of using both shaderc and glslc and an example of using GLSL and HLSL and this was the simplest way to do it :)

Do you intend to do this refactoring? (I'm very happy to merge this in the meantime, just bump the changelog and version number (after rebasing))

ukari · 2021-02-19T19:18:30Z

now the structure

exposed-modules:
      Vulkan.Utils.ShaderQQ.Backend.Glslang (GlslangError, GlslangWarning, processGlslangMessages)
      Vulkan.Utils.ShaderQQ.Backend.Shaderc (ShadercError, ShadercWarning, processShadercMessages)
      Vulkan.Utils.ShaderQQ.GLSL.Glslang (glsl, vert .. mesh, compileShaderQ, compileShader)
      Vulkan.Utils.ShaderQQ.GLSL.Shaderc (glsl, vert .. mesh, compileShaderQ, compileShader)
      Vulkan.Utils.ShaderQQ.HLSL.Glslang (hlsl, vert .. mesh, compileShaderQ, compileShader)
      Vulkan.Utils.ShaderQQ.HLSL.Shaderc (hlsl, vert .. mesh, compileShaderQ, compileShader)
  other-modules:
      Vulkan.Utils.ShaderQQ.ShaderType
      Vulkan.Utils.ShaderQQ.Backend.Glslang.Internal
      Vulkan.Utils.ShaderQQ.Backend.Shaderc.Internal
      Vulkan.Utils.ShaderQQ.GLSL
      Vulkan.Utils.ShaderQQ.HLSL

about `Vulkan.Utils.ShaderQQ.HLSL.Glslang`

when compile a hlsl shader, glslangValidator needs to pass a entry point name explicitly while glslc could auto detect entry point.

frag.hlsl

float4 main([[vk::location(0)]] const float3 col) : SV_TARGET
{
    return float4(col, 1);
}

glslangValidator with entry point name

$ glslangValidator -D -S frag -V test.frag -e main --target-env spirv1.0 -o frag.spv
test.frag
$ spirv-cross frag.spv --reflect
{
    "entryPoints" : [
        {
            "name" : "main",
            "mode" : "frag"
        }
    ],
    "inputs" : [
        {
            "type" : "vec3",
            "name" : "col",
            "location" : 0
        }
    ],
    "outputs" : [
        {
            "type" : "vec4",
            "name" : "@entryPointOutput",
            "location" : 0
        }
    ]
}

glslangValidator without entry point name

$ glslangValidator -D -S frag -V test.frag --target-env spirv1.0 -o frag.spv
test.frag
WARNING: Linking fragment stage: Entry point not found
$ spirv-cross frag.spv --reflect
{
    "entryPoints" : [
        {
            "name" : "",
            "mode" : "frag"
        }
    ]
}

glslc without pass entry point name

$ glslc -fshader-stage=frag -x hlsl --target-spv=spv1.0 test.frag -o frag.spv
$ spirv-cross frag.spv --reflect
{
    "entryPoints" : [
        {
            "name" : "main",
            "mode" : "frag"
        }
    ],
    "inputs" : [
        {
            "type" : "vec3",
            "name" : "col",
            "location" : 0
        }
    ],
    "outputs" : [
        {
            "type" : "vec4",
            "name" : "@entryPointOutput",
            "location" : 0
        }
    ]
}

ref: glslangValidator Complete HLSL -> SPIR-V translator

I thought that if glslc don't need pass the entry point name and could auto detect entry name, it should be possible for glslangValidator to do the same thing without pass the entry point name. But now glslangValidator can't do this.

In haskell spec, quoter cannot be an arbitrary expression. It means syntax like [frag "main"| ... |] is impossible.

A quasi-quote has the form [quoter| string |].

    The ⟨quoter⟩ must be the name of an imported quoter, either qualified or unqualified; it cannot be an arbitrary expression.

Is there any advice for the design of vert, frag ... mesh in Vulkan.Utils.ShaderQQ.HLSL.Glslang? Or just remove glslang QuasiQuote support for hlsl and just provide compileShaderQ, compileShader with argument for passing entry point name in Vulkan.Utils.ShaderQQ.HLSL.Glslang?

dpwiz · 2021-02-20T07:40:38Z

A QuasiQuoter can generate a continuation that will accept more data and finish the generation.

ukari · 2021-02-20T16:12:15Z

now the usage for vert ... mesh in Vulkan.Utils.ShaderQQ.HLSL.Glslang has changed to syntax like the following example.

{-# LANGUAGE QuasiQuotes #-}
{-# LANGUAGE TemplateHaskell #-}

let vertCode = $([vert|
    ...
    VSOutput main(const uint i : SV_VertexID)
        {
            VSOutput output;
            output.pos = float4(positions[i], 0, 1.0);
            output.col = colors[i];
            return output;
         }
    |] "main")

here is another syntax use MonadIO ByteString as the result type instead of use $() to expand Q Exp to ByteString in template haskell. I have implement this syntax locally but in this implement the hlsl is not compiled with source location to prevent use Q ByteString as result type.

{-# LANGUAGE QuasiQuotes #-}
vertCode <- [vert|
  ...
  VSOutput main(const uint i : SV_VertexID)
      {
          VSOutput output;
          output.pos = float4(positions[i], 0, 1.0);
          output.col = colors[i];
          return output;
      }
  |] "main"

which is better?

expipiplus1 · 2021-02-21T04:27:09Z

I think to be honest I'd just leave it as hardcoded to "main" for the quasiquoter. The quasiquoters are intended to be a quick convenience and anything more involved (like multiple shaders in one file with different entrypoints) deserve to be files in their own right, or the user can fall back to compileShaderQ

ukari · 2021-02-21T06:59:45Z

squashed, please review it

expipiplus1 · 2021-02-23T04:48:49Z

Looks good to me, thanks!

Happy to merge if the compile error is fixed. Could you also add some details to the changelog please? If you want this in a release sooner then feel free to bump the version of vulkan-utils!

ukari · 2021-02-24T08:49:47Z

actually I am not sure which version for vulkan-utils should be used for changelog

expipiplus1 · 2021-02-24T08:54:11Z

Either in the WIP section or 0.5 if you want a release more quickly.

…

On Wed, Feb 24, 2021, 4:50 PM 漁囿 ×.̷.̷.̷ •ゞ潅頭 àｉ < ***@***.***> wrote: actually I am not sure which version for vulkan-utils should be used for changelog — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#263 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGRJXGASIBP2NOA77EXJ23TAS4TTANCNFSM4XYBN47Q> .

- refactor shaderQQ - update changelog - update comments - specify --target-spv for shaderc ray tracing shaders - specify --target-env for glslang ray tracing shaders - support pass hlsl entry point to glslangValidator and shaderc - support pass glsl entry point to glslangValidator - support compile both glsl and hlsl with both glslang and shaderc

ukari · 2021-02-25T11:06:38Z

update & squashed, please review again

expipiplus1 · 2021-02-27T09:38:07Z

Looks like a couple of the tests are failing because of changed line numbers. Is it possible to fix these please, then happy to merge?

ukari · 2021-02-27T19:01:33Z

Thanks for your review.

the cause

I check some github ci log and found that this is caused by the lack of shaderc. I changed the some examples in vulkan/examples to make sure all shaderQQ modules could be checked to be compile successful which means all modules of HLSL.Shaderc, HLSL.Glslang, GLSL.Shaderc, GLSL.Glslang are used in vulkan/examples.

the usage situation of all modules of HLSL.Shaderc, HLSL.Glslang, GLSL.Shaderc, GLSL.Glslang in vulkan/examples.

$ grep -Ri "ShaderQQ" *
compute/Main.hs:import           Vulkan.Utils.ShaderQQ.GLSL.Glslang
hlsl/Pipeline.hs:import           Vulkan.Utils.ShaderQQ.HLSL.Glslang ( vert )
hlsl/Pipeline.hs:import           Vulkan.Utils.ShaderQQ.HLSL.Shaderc ( frag )
offscreen/Main.hs:import           Vulkan.Utils.ShaderQQ.GLSL.Glslang
rays/Pipeline.hs:import           Vulkan.Utils.ShaderQQ.GLSL.Glslang  ( glsl )
rays/Pipeline.hs:import qualified Vulkan.Utils.ShaderQQ.GLSL.Glslang as Glslang
rays/Pipeline.hs:import qualified Vulkan.Utils.ShaderQQ.GLSL.Shaderc as Shaderc
readme.md:- Convenient shader creation using the `Vulkan.Utils.ShaderQQ.comp` QuasiQuoter
resize/Pipeline.hs:import           Vulkan.Utils.ShaderQQ.GLSL.Glslang
resize/Julia.hs:import           Vulkan.Utils.ShaderQQ.GLSL.Glslang
sdl-triangle/Main.hs:import           Vulkan.Utils.ShaderQQ.GLSL.Glslang ( frag )
sdl-triangle/Main.hs:import           Vulkan.Utils.ShaderQQ.GLSL.Shaderc ( vert )

the dependency on shaderc/glslc in vulkan/examples.

$ grep -Ri "ShaderQQ.*.Shaderc" *
hlsl/Pipeline.hs:import           Vulkan.Utils.ShaderQQ.HLSL.Shaderc ( frag )
rays/Pipeline.hs:import qualified Vulkan.Utils.ShaderQQ.GLSL.Shaderc as Shaderc
sdl-triangle/Main.hs:import           Vulkan.Utils.ShaderQQ.GLSL.Shaderc ( vert )

For example, I import GLSL.Glslang(frag) and GLSL.Shaderc(vert) in examples/sdl-triangle/Main.hs. This cause the ci fails when compile the use of vert due to the dependency lack of shaderc/glslc.

why some ci successfully build

In the current version of expipiplus1/vulkan, there is a have-shaderc flag to help skip the build of examples/hlsl to prevent the need of shaderc/glslc.

the use of have-shaderc in ci.yml

    - run: |
        if command -v glslc &> /dev/null; then
          cabal configure -fgeneric-instances -f-raytracing -fhave-shaderc
        else
          echo "glslc could not be found"
          cabal configure -fgeneric-instances -f-raytracing -f-have-shaderc
        fi
      shell: bash

Some ci build passed successfully because of the build of examples is skipped due to their ghc version is 8.6.5.
remove examples for ghc 8.6.5

    - name: Remove examples for 8.6.5
      run: |
        sed -ibak '/examples/d' cabal.project
      if: matrix.ghc == '8.6.5'

It seems only in the Windows OS, the examples/hlsl(which needs shaderc/glslc) is compiled. (I didn't check this exactly)

the solution

the revert solution (which I choose)

change vulkan/examples back to the current upstream version, so that only examples/hlsl needs shaderc/glslc.

the complex solution

update .github/workflows/ci.yml to provide shaderc/glslc in ci environment
update examples/readme.md

ukari · 2021-02-27T20:01:42Z

I didn't realize the existence of doctest before I run stack test just now.

now doctest failures has been fixed by changed the line number to the same as doctest expects.

expipiplus1 · 2021-03-01T08:14:30Z

amazing, thanks

expipiplus1 · 2021-03-01T09:20:10Z

released: https://hackage.haskell.org/package/vulkan-utils-0.5.0

thanks again.

expipiplus1 and others added 2 commits February 27, 2021 14:22

Merge branch 'master' into master

182c8c3

Correct Haddock parse error

ecd1b99

ukari added 2 commits February 28, 2021 03:13

update examples to make sure only example hlsl needs glslc

c5779b4

fix line number for doctest

ede03ec

expipiplus1 merged commit 74dc6b4 into expipiplus1:master Mar 1, 2021

ukari mentioned this pull request Apr 3, 2021

WIP: Add SPIR-V reflect to vulkan-utils #270

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support more shader stages #263

support more shader stages #263

ukari commented Feb 17, 2021

expipiplus1 commented Feb 18, 2021

ukari commented Feb 18, 2021 •

edited

Loading

ukari commented Feb 18, 2021

expipiplus1 commented Feb 19, 2021 •

edited

Loading

ukari commented Feb 19, 2021

dpwiz commented Feb 20, 2021

ukari commented Feb 20, 2021 •

edited

Loading

expipiplus1 commented Feb 21, 2021

ukari commented Feb 21, 2021

expipiplus1 commented Feb 23, 2021

ukari commented Feb 24, 2021

expipiplus1 commented Feb 24, 2021 via email

ukari commented Feb 25, 2021

expipiplus1 commented Feb 27, 2021

ukari commented Feb 27, 2021

ukari commented Feb 27, 2021

expipiplus1 commented Mar 1, 2021

expipiplus1 commented Mar 1, 2021

support more shader stages #263

support more shader stages #263

Conversation

ukari commented Feb 17, 2021

expipiplus1 commented Feb 18, 2021

ukari commented Feb 18, 2021 • edited Loading

ukari commented Feb 18, 2021

expipiplus1 commented Feb 19, 2021 • edited Loading

ukari commented Feb 19, 2021

now the structure

about Vulkan.Utils.ShaderQQ.HLSL.Glslang

dpwiz commented Feb 20, 2021

ukari commented Feb 20, 2021 • edited Loading

expipiplus1 commented Feb 21, 2021

ukari commented Feb 21, 2021

expipiplus1 commented Feb 23, 2021

ukari commented Feb 24, 2021

expipiplus1 commented Feb 24, 2021 via email

ukari commented Feb 25, 2021

expipiplus1 commented Feb 27, 2021

ukari commented Feb 27, 2021

the cause

why some ci successfully build

the solution

the revert solution (which I choose)

the complex solution

ukari commented Feb 27, 2021

expipiplus1 commented Mar 1, 2021

expipiplus1 commented Mar 1, 2021

ukari commented Feb 18, 2021 •

edited

Loading

expipiplus1 commented Feb 19, 2021 •

edited

Loading

about `Vulkan.Utils.ShaderQQ.HLSL.Glslang`

ukari commented Feb 20, 2021 •

edited

Loading