How to include GL_ extensions?

[JonathanWoollett-Light]

Trying to write the line:

uint tester0 = gl_SubgroupSize;

In a GLSL shader results in:

error: 'gl_SubgroupSize' : required extension not requested: GL_KHR_shader_subgroup_basic

Minimal reproducible example: vulkano-test.zip

I've looked through some of the documentation and I can't figure out how to fix this.

I would greatly appreciate any help.

[Eliah-Lakhin]

@JonathanWoollett-Light You can read subgroup size outisde of the shader using Vulkano API and then pass the value to the shader through specialization constants/uniforms etc. See this example: https://github.com/vulkano-rs/vulkano/blob/master/examples/src/bin/dynamic-local-size.rs#L119 Moreover, it shows how to configure compute shader's local size(in case it was your actual goal).

[JonathanWoollett-Light]

@Eliah-Lakhin In doing that I can see how I could get the values I need. Passing a buffer of subrgoupIDs and such.

But this feels aweful, it feels like passing such an amount of unneccessary data, do you know why in this circumstance the functionality is unavailable when in a bunch of other GLSL code examples it is available? Is it a lacking of vulkano?

[AustinJ235]

you could use specialization constants also either at pipeline creation or at the dispatch call ⁣Get BlueMail for Android ​

…

On Jul 8, 2020, 7:24 PM, at 7:24 PM, Jonathan Woollett-Light ***@***.***> wrote: In doing that I can see how I could get the values I need. Passing a buffer of subrgoupIDs and such. But this feels aweful, it feels like passing such an amount of unneccessary data, do you know why in this circumstance the functionality is unavailable when in a bunch of other GLSL code examples it is available? Is it a lacking of vulkano? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #1395 (comment)

[Eliah-Lakhin]

@JonathanWoollett-Light No, no, passing specialization constants is completely free(in contrast to buffers). They are inline by a GPU when it compiles spirv bytecode into it's internal assembler. This approach is a bit verbose, but that's how Vulkan designed. In fact Vulkano simplifies a lot of Vulkan API verbosity under the hood.

[Eliah-Lakhin]

@JonathanWoollett-Light I just tried to fix your snippet. Vulkan documentation saying that a specific extension needs to be enabled in glsl code. To be more specific for this particular function you need to add #extension GL_KHR_shader_subgroup_basic: enable:

        #version 450

        #extension GL_KHR_shader_subgroup_basic: enable

        layout(local_size_x = 64, local_size_y = 1, local_size_z = 1) in;

        layout(set = 0,binding = 0) buffer Buffer {
            uint x[];
        };

        void main() {
            uint indx = gl_GlobalInvocationID.x;
            uint tester0 = gl_SubgroupSize;
            uint tester1 = gl_NumSubgroups;
        }"

However, it doesn't work too. Vulkano shader compiler(that uses Google's shaderc under the hood) throughs an error: "ParseError(UnknownConstant("Capability", 61))".

I would say this seem to be a lack of Vulkano capabilities. Or maybe even Shaderc compiler. May I ask you to fix this? The shader compiler code resides here: https://github.com/vulkano-rs/vulkano/tree/master/vulkano-shaders/src

[JonathanWoollett-Light]

@Eliah-Lakhin So Vulkano (or shaderc?) doesn't support #extension in GLSL? Or is it more specifically lack of support for GL_KHR_shader_subgroup_basic?

There are many things I will need to learn if I'm to fix this. It is unfortunate I have to.

This seems like a very signficant thing to not work.

[JonathanWoollett-Light]

@AustinJ235 Worth mentioning doing this approach is unfeasible for implementation of a scan compute shader (e.g. summing a vector) as it requires accessing gl_SubgroupID and gl_SubgroupInvocationID (and other IDs) which would be immensely awkward to pass this way (I think it would also be impossible to implement scan like this, but I have not tried).

It's a very significant issue.

[JonathanWoollett-Light]

@Eliah-Lakhin @Eliah-Lakhin

So just to come back to this after a while...

It certainly is possible to compile shaders using shaderc which make use of subgroups.

So I think the bug likely lies within Vulkano.

An example of compiling a shader using subgroups with shaderc's glslc:

C:/VulkanSDK/1.2.148.1/Bin/glslc C:/Users/jonat/Projects/test-project/glsl/atomicArraySum.comp -o C:/Users/jonat/Projects/test-project/spir-v/atomicArraySum.spv --target-env=vulkan1.1

Importantly note --target-env=vulkan1.1, so perhaps Vulkano is using --target-env=vulkan1.0 which may be causing the error?

[AustinJ235]

vulkano targets 1.1 currently. I don't think it possible to use opengl extensions could be wrong though.

…

On Oct 15, 2020, 9:32 PM, at 9:32 PM, Jonathan Woollett-Light ***@***.***> wrote: @Eliah-Lakhin @Eliah-Lakhin So just to come back to this after a while... It certainly is possible to compile shaders using shaderc which make use of subgroups. So I think the bug likely lies within Vulkano. An example of compiling a shader using subgroups with shaderc's glslc: ``` C:/VulkanSDK/1.2.148.1/Bin/glslc C:/Users/jonat/Projects/test-project/glsl/atomicArraySum.comp -o C:/Users/jonat/Projects/test-project/spir-v/atomicArraySum.spv --target-env=vulkan1.1 ``` Importantly note `--target-env=vulkan1.1`, so perhaps Vulkano is using `--target-env=vulkan1.0` which may be causing the error? -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #1395 (comment)

[JonathanWoollett-Light]

I don't think it possible to use opengl extensions

I'm not sure what you mean by this.

Certainly such extensions can be used for Vulkan enviroments, a quote from some Vulkan docs:

Alongside the Vulkan 1.1 core subgroup functionality, a new GLSL extension GL_KHR_shader_subgroup which is available here, and is usable in glslang too.

Shaders can be compiled and used with subgroup operations using --target-env=vulkan1.1. For example I used such a shader with wgpu-rs (and this was specifically using the Vulkan enviroment) (worth mentioning here there are some questionable things going on, but the shader does work with vulkan).

I think the issue must be specific to Vulkano.

[Eliah-Lakhin]

@JonathanWoollett-Light I agree, it probably should be possible to use some of the GLSL extensions that compatible with Vulkan. I didn't have a chance to work on such extensions support, so can't say much. I myself tried to include one of them a while ago, but seem like the current Vulkan-Shaders subproject is not capable to manage them properly, but I think it should be possible to improve it to use them.

Would you like to work on these improvements. If Vulkan version target is an issue, don't hesitate to raise it whenever it needed. We will be glad to accept your contribution to the project.

[JonathanWoollett-Light]

@Eliah-Lakhin

Would you like to work on these improvements. If Vulkan version target is an issue, don't hesitate to raise it whenever it needed. We will be glad to accept your contribution to the project.

At the moment attempting to make such fixes is quite beyond my knowledge. I know practically nothing about compilation of shaders and the surrounding topics.

However I would be willing to learn if you have any suggestions for where I might start looking.

Do you know any good resources on or around the subject?

[Eliah-Lakhin]

@JonathanWoollett-Light Well, in Vulkano Shaders we actually don't compile glsl code ourself, instead we user Shaderc-rs crate that does it for us. And in Vulkano Shaders we only configuring their compiler, and then analyzing metadata received from the Compiler. So it's all not so complex.

[JonathanWoollett-Light]

So I have quite a few questions:

Question 1

My first point of confusion arises at:

const ENV_VULKAN_VERSION: u32 = (1 << 22) | (1 << 12);

From the function:

pub fn compile(
    path: Option<String>,
    base_path: &impl AsRef<Path>,
    code: &str,
    ty: ShaderKind,
    include_directories: &[impl AsRef<Path>],
    macro_defines: &[(impl AsRef<str>, impl AsRef<str>)],
) -> Result<CompilationArtifact, String> {
    let mut compiler = Compiler::new().ok_or("failed to create GLSL compiler")?;
    let mut compile_options = CompileOptions::new().ok_or("failed to initialize compile option")?;
    const ENV_VULKAN_VERSION: u32 = (1 << 22) | (1 << 12);
    compile_options.set_target_env(TargetEnv::Vulkan, ENV_VULKAN_VERSION);
    let root_source_path = if let &Some(ref path) = &path {
        path
    } else {
        // An arbitrary placeholder file name for embedded shaders
        "shader.glsl"
    };

    // Specify file resolution callback for the `#include` directive
    compile_options.set_include_callback(
        |requested_source_path, directive_type, contained_within_path, recursion_depth| {
            include_callback(
                requested_source_path,
                directive_type,
                contained_within_path,
                recursion_depth,
                include_directories,
                path.is_some(),
                base_path,
            )
        },
    );

    for (macro_name, macro_value) in macro_defines.iter() {
        compile_options.add_macro_definition(macro_name.as_ref(), Some(macro_value.as_ref()));
    }

    let content = compiler
        .compile_into_spirv(&code, ty, root_source_path, "main", Some(&compile_options))
        .map_err(|e| e.to_string())?;

    Ok(content)
}

So, this sets the value of EN_VULKAN_VERSION to 4198400.

1<<22=000000010000000000000000000000
1<<12=000000000000000001000000000000

000000010000000000000000000000 |
000000000000000001000000000000 =
000000010000000001000000000000 = 4198400

Why is the version set like this? (It seems such an incredibly awkward format for the version)

Question 2

Looking at the line let mut compiler = Compiler::new().ok_or("failed to create GLSL compiler")?; in codegen::compile.

When going to the definition of Compiler I see:

pub struct Compiler {
    raw: *mut scs::ShadercCompiler,
}

So it is just a wrapper, but when looking at the definition of ShadercCompiler I see:

pub enum ShadercCompiler {}

This seems useless what am I missing here? (I imagine it is some functionality of how libc works that I do not understand)

Question 2.1

Really the same as the question 2 but just repeated for let mut compile_options = CompileOptions::new().ok_or("failed to initialize compile option")?; into CompileOptions into scs::ShadercCompileOptions into pub enum ShadercCompileOptions {}.

[Rua]

I think this issue is no longer relevant now?