[wgsl] Textures

[dj2]

The design for textures needs to be added to the spec. Need to determine a good syntax for textures in the language.

[dj2]

Looking at various shading languages:

MSL

enum class access { sample, read, write, read_write };
texture1d<T, access a = access::sample>
texture1d_array<T, access a = access::sample>
texture2d<T, access a = access::sample>
texture2d_array<T, access a = access::sample>
texture3d<T, access a = access::sample>
texturecube<T, access a = access::sample>
texturecube_array<T, access a = access::sample>
texture2d_ms<T, access a = access::read>
texture2d_ms_array<T, access a = access::read>  # OSX Metal 2.0, IOS ???

depth2d<T, access a = access::sample>
depth2d_array<T, access a = access::sample>
depthcube<T, access a = access::sample>
depthcube_array<T, access a = access::sample>
depth2d_ms<T, access a = access::read>
depth2d_ms_array<T, access a = access::read>  # OSX Metal 2.0, IOS ???

For texture types
(except depth texture types), T can be half, float, short, ushort, int, or uint. For depth texture types, T must be float.

HLSL

Object1 [<Type>] Name;

Object1 Type | Description
-- | --
Buffer
Texture1D
Texture1DArray
Texture2D
Texture2DArray
Texture3D
TextureCube
TextureCubeArray


Object2 [<Type, Samples>] Name;

Object2 Type | Description
-- | --
Texture2DMS
Texture2DMSArray

Type | Optional. Any scalar HLSL type or vector HLSL type, surrounded by angle brackets. The default type is float4.

I'm not sure what the depth equivalents would be in HLSL.

GLSL

gsampler1D 
gsampler2D
gsampler3D
gsamplerCube
gsampler2DRect
gsampler1DArray
gsampler2DArray
gsamplerCubeArray  # requires GL 4.0 or ARB_texture_cube_map_array
gsamplerBuffer
gsampler2DMS
gsampler2DMSArray

For `float` the `g` goes away, for `integer` the `g` becomes an `i` and for `unsigned` the `g` becomes a `u`

WGSL

Given the above I'd propose something like the following for WGSL.

texture_1d<type>
texture_1d_array<type>
texture_2d<type>
texture_2d_array<type>
texture_3d<type>
texture_cube<type>
texture_cube_array<type>
texture_2d_ms<type, size>
texture_2d_ms_array<type, size>

[Kangz]

I think you forgot SPIR-V and the "shadow samplers" in GLSL

[dj2]

GLSL Shadow

sampler1DShadow
sampler2DShadow
samplerCubeShadow
sampler2DRectShadow
sampler1DArrayShadow
sampler2DArrayShadow
samplerCubeArrayShadow

Does this map to the depth samplers in MSL? From [1]:

If a texture has a depth or depth-stencil image format and has the depth comparison activated, it cannot be used with a normal sampler. Attempting to do so results in undefined behavior. Such textures must be used with a shadow sampler.

1. https://www.khronos.org/opengl/wiki/Sampler_(GLSL)#Shadow_samplers

[dj2]

For the SPIR-V side, there is a single OpTypeImage which takes a series of parameters:

Sampled Type | Dim | Depth | Arrayed | MS | Sampled | Image Format | Optional Access Qualifier

Sampled Type is the type of the components that result from sampling or reading from this image type. Must be a scalar numerical type or OpTypeVoid.

Depth is whether or not this image is a depth image. (Note that whether or not depth comparisons are actually done is a property of the sampling opcode, not of this type declaration.)
0 indicates not a depth image
1 indicates a depth image
2 means no indication as to whether this is a depth or non-depth image

Arrayed must be one of the following indicated values:
0 indicates non-arrayed content
1 indicates arrayed content

MS must be one of the following indicated values:
0 indicates single-sampled content
1 indicates multisampled content

Sampled indicates whether or not this image will be accessed in combination with a sampler, and must be one of the following values:
0 indicates this is only known at run time, not at compile time
1 indicates will be used with sampler
2 indicates will be used without a sampler (a storage image)

Image Format is the Image Format, which can be Unknown, as specified by the client API.

[austinEng]

Adding this relevant note which surprised me when working with depth sampling. In Vulkan, "Validation Rules within a Module"

The “Depth” operand of OpTypeImage is ignored.

[dj2]

Does that mean depth sampling doesn't work in Vulkan, or it doesn't care what you say and lets you do it no matter what you set?

[austinEng]

From what I understand, depth sampling does work and it simply ignores the Depth operand. The spec is very unclear about whether or not the image passed to ImageSampleDrefImplicitLod need be bound to an actual depth texture.

Metal's documentation is also not clear about this.

In HLSL it is valid to comparison-sample a color texture.
https://docs.microsoft.com/en-us/windows/win32/direct3dhlsl/dx-graphics-hlsl-to-samplecmp

The comparison is a single component comparison between the first component stored in
the texture, and the comparison value passed into the method.

Experimentally, neither Metal nor Vulkan's validation layers complain if you depth-sample a color texture, though there's inconsistent behavior depending on the texture format / driver. Sometimes it follows the HLSL semantics and does the depth-comparison with the first component, sometimes it simply returns the value of the first component, and sometimes it returns something else.

[kvark]

Thank you for investigation @dj2 and @austinEng !

MSAA

texture_2d_ms<type, size>
texture_2d_ms_array<type, size>

I don't think we need the size here. Haven't personally seen it specified in HLSL. Also, here is something I found on the internets:

As can be seen in these declarations, the multisampled texture resource objects spec-
ify their format and can also optionally specify the number of subsamples that are used in
the resource. This sample count used to be required in Direct3D 10, but since Direct3D
10.1, the count specification has become optional. Now it is possible to query the number
of samples through the HLSL GetDimensions() method, making the direct specification
unnecessary. This ensures that any methods that access the subsamples only try to use the
appropriate number of samples. The semantics surrounding the usage of each of these re-
source objects will be covered in more detail in Chapter 6.

It would be highly inconvenient for the users if we require it. More often than not, the decision on MSAA quality is made at run-time, and not everybody wants to unroll the MSAA iteration loops and generate separate shaders per sample count.

Depth comparison

Let's not try to do depth comparison on color values. That's not terribly useful, hard to guarantee, and complicates the grammar/shader language.

It looks like Metal requires us to specify if the texture is only used for depth comparison sampling...
@dj2 do you have a proposal for how we are going to express that in WGSL?

[dj2]

Not yet, was waiting to see if someone could point me to how the MSL depth and GLSL shadow map to HLSL. From the first 2, it seems like we should have a specific word and I'd lean towards how MSL does it as it's clearer what depth means then shadow.

[kvark]

In HLSL you have a regular Texture2D and sample it with SampleCmp and SampleCmpLevelZero. Note that HLSL doesn't like 2DMS/3D textures in there:

Any texture-object type (except Texture2DMS, Texture2DMSArray, or Texture3D).

[dj2]

So, then the intersection of those 3 APIs seems to be:

WGSL

depth_2d<type>
depth_2d_array<type>
depth_cube<type>
depth_cube_array<type>

Unless I missed something?

[litherum]

WebGPU currently supports 1d textures, cube textures, and 3d textures. It looks like all 3 APIs support those.

[dneto0]

On Vulkan depth textures. My understanding is that it's all controlled from the API side, and only certain image formats are compatible with being attached as a depth texture. See documentation related to VK_FORMAT_FEATURE_DEPTH_STENCIL_ATTACHMENT_BIT.

[dneto0]

As for the proposal, we need to make clear what the type parameter is. I presume it's the sampled type, i.e. what is the component type of the data you get back when sampling.

Depending on what operations are being performed on an image, Vulkan sometimes requires the shader to declare the image format. E.g. if doing image atomics. That will affect the declaration, I think. I'll need to look further.

[grorg]

Discussed at the 2020-06-09 meeting.

[kvark]

I see in MSL spec that depth textures provide both sample() and sample_compare() methods. It's not clear to me whether you can do sample() on a sampler that has comparison function set up. This seems non-portable to other APIs.
Our samplers in WGSL need to know if they are comparison samplers or not (since the WebGPU API makes this distinction in GPUBindGroupLayoutEntry). So I think we could have very similar sample() and sample_compare() semantics in WGSL, enforced by the sampler and texture types (e.g. in a.sampler_compare(b, ..), a has the type of a depth texture, and b has the type of a comparison sampler). That seems portable to HLSL and SPIR-V.

[dj2]

Thinking of samplers and comparison samplers, ignoring depth and stencil for now.

Declare Texture

texture_1d<type>
texture_2d<type>
texture_3d<type>
texture_2d_ms<type>

• type must be f32, i32 or u32
• The parameterized type for the images is the type after conversion from reading/sampling.
  E.g. you can have an image with texels with 8bit unorm components, but when you sample
  them you get a 32-bit float result (or vec-of-f32).

%1 = OpTypeImage %type <1|2|3> 0 0 <0|1 for ms> 1 Unknown
(HLSL: Texture2D : register(t1))
(MSL: texture2d [[texture(0)]])
(GLSL: texture2D)

Declare Sampler

sampler

%14 = OpTypeSampler
(HLSL: SamplerState : register(s2))
(MSL: sampler [[sampler(0)]])
(GLSL: sampler)

comparison_sampler

OpTypeSampler
(HLSL: SamplerComparisonState)
(MSL: sampler)
(GLSL: samplerShadow)

Fetch texel from texture

vec4<type> my_texture.load(<sampler>, [i32 | vec(2|3)<i32>(coords)], i32 level_of_detail)

%32 = OpImageFetch %v4float %31 %29 Lod %int_2
(HLSL: my_texture.Load(int3(coords, lod)))
(MSL: my_texture.read(uint2(coords), lod))
(GLSL: texelFetch(sampler2D(my_texture, sampler), coords, lod))

Sample from a texture

vec4<type> my_texture.sample(<sampler>, i32 | vec(2|3) coords)

%24 = OpImageSampleImplicitLod %v4float %19 %23
(HLSL: my_texture.Sample(sampler, coord))
(MSL: my_texture.sample(sampler, coord))
(GLSL: texture(sampler2D(my_texture, sampler), coords))

vec4<type> my_texture.sample(<sampler>, i32 | vec(2|3) coords, i32 lod)
%25 = OpImageSampleExplicitLod %v4float %19 %23 Lod %float_0
(HLSL: my_texture.SampleLevel(sampler, coords, lod))
(MSL: my_texture.sample(sampler, coords, lod))
(GLSL: textureLod(sampler2D(my_texture, sampler), coords, lod))

Comparison sample from a texture

f32 = texture.sample_compare(comparison_sampler, i32 | vec(2|3) coords, i32 bias)
%65 = OpImageSampleDrefImplicitLod %float %62 %63 %64
(HLSL: texture.SampleCmp(comparison_sampler, coords, bias))
(MSL: texture.sample_compare(comparison_sampler, coords, bias))
(GLSL: texture(sampler2DShadow(texture, comparison_sampler), coords, bias))

I believe the only grammar changes needed are the ability to declare the textures and samplers. The calls look like method calls, so we'd just need to validate that the method called on the texture is correct during validation.

[kvark]

@dj2 thank you for writing this down!

Thinking of samplers and comparison samplers, ignoring depth and stencil for now.

I don't think you can talk about comparison samplers without considering depth textures. IIRC, the limitations of the API force us to only use one with another. Using a comparison sampler with a regular texture is not portable to MSL, for example.

vec4 my_texture.load(, [i32 | vec(2|3)(coords)], i32 level_of_detail)

Why does it get a <sampler> as the first parameter?

vec4 my_texture.sample(, i32 | vec(2|3) coords, i32 lod)

Can we please restrict the overloading semantics of functions to only be within "generics"? Basically, it makes sense to have functions polymorphic of some types, but I don't think we need to extend this to calling the same function with different set of parameters. It introduces ambiguity, confusion, and doesn't really get us much over the explicit variant:

vec4<type> my_texture.sample_level(<sampler>, i32 | vec(2|3) coords, i32 lod)

vec4 my_texture.sample(, i32 | vec(2|3) coords)

This should be f32 and the vec2/3 should be of ` specifically.

Also, cube arrays would require vec4 coordinates.

f32 = texture.sample_compare(comparison_sampler, i32 | vec(2|3) coords, i32 bias)

MSL doesn't allow regular textures to do sample_compare, only depth textures can do this.

Other questions

• Does this only describe sample-able textures?
• Can a texture bound as readonly-storage-texture be sampled? MSL, for example, doesn't allow that.
• Where do we provide the exact format for storage textures?

[dj2]

Thinking of samplers and comparison samplers, ignoring depth and stencil for now.

I don't think you can talk about comparison samplers without considering depth textures. IIRC, the limitations of the API force us to only use one with another. Using a comparison sampler with a regular texture is not portable to MSL, for example.

I think that's taken into account with the methods taking either a sampler or a comparison sampler?

vec4 my_texture.load(, [i32 | vec(2|3)(coords)], i32 level_of_detail)

Why does it get a <sampler> as the first parameter?

As far as I can tell, with SPIR-V you need to have the sampler as it is part of the OpImageFetch call

         %10 = OpTypeImage %float 2D 0 0 0 1 Unknown
         %14 = OpTypeSampler
         %18 = OpTypeSampledImage %10
         %13 = OpLoad %10 %myTexture
         %17 = OpLoad %14 %mySampler
         %19 = OpSampledImage %18 %13 %17
         %31 = OpImage %10 %19
         %32 = OpImageFetch %v4float %31 %29 Lod %int_2

GLSL appears to be similar in that you end up doing sampler2D(texture, sampler)

vec4 my_texture.sample(, i32 | vec(2|3) coords, i32 lod)

Can we please restrict the overloading semantics of functions to only be within "generics"? Basically, it makes sense to have functions polymorphic of some types, but I don't think we need to extend this to calling the same function with different set of parameters. It introduces ambiguity, confusion, and doesn't really get us much over the explicit variant:

vec4<type> my_texture.sample_level(<sampler>, i32 | vec(2|3) coords, i32 lod)

Works for me.

vec4 my_texture.sample(, i32 | vec(2|3) coords)

This should be f32 and the vec2/3 should be of ` specifically.

Also, cube arrays would require vec4 coordinates.

There appears to be a bit of missing text here, I'm guessing you mean should be f32 specifically? So, this should say f32 | vec(2|3|4)<f32>)?

f32 = texture.sample_compare(comparison_sampler, i32 | vec(2|3) coords, i32 bias)

MSL doesn't allow regular textures to do sample_compare, only depth textures can do this.

Oh, missed that when reading through the MSL code. So, then we do need to add the:

Depth textures

depth_2d<type>
depth_cube<type>

%1 = OpTypeImage %type <1|2|3> 0 0 <0|1 for ms> 1 Unknown

%53 = OpTypeImage %float <2|Cube> 1 0 <1|0 for ms> Unknown
(HLSL: Texture2D : register(t1))
(MSL: depth2d [[texture(0)]])
(GLSL: texture2D)

Other questions

• Does this only describe sample-able textures?

Yes.

• Can a texture bound as readonly-storage-texture be sampled? MSL, for example, doesn't allow that.

If MSL doesn't allow it, then I don't think we can do it unless there is some way to work around it in MSL.

• Where do we provide the exact format for storage textures?

Haven't gotten there yet, do you have a GLSL example where this is done?

[dj2]

vec4<type> my_texture.load(<sampler>, [i32 | vec(2|3)<i32>(coords)], i32 level_of_detail) does not need a sampler. I failed to understand how SPIR-V worked, so this would be vec4<type> my_texture.load([i32 | vec(2|3)<i32>(coords)], i32 level_of_detail) for the load

[dj2]

Change from method call syntaxes to functions:

• vec4<type> texture_load(<texture>, [i32 | vec(2|3)<i32>(coords)], i32 level_of_detail)
• vec4<type> texture_sample(<texture>, <sampler>, i32 | vec(2|3) coords)
• vec4<type> texture_sample_level(<texture>, <sampler>, i32 | vec(2|3) coords, i32 lod)
• f32 texture_sample_compare(<depth_texture>, <comparison_sampler>, i32 | vec(2|3) coords, i32 bias)

[grorg]

Discussed at the 2020-06-24 WebGPU Virtual F2F.

[kdashg]

Relevant: #532 (comment)

[kvark]

@dj2 a few corrections are needed

1. all the non-load texture coordinates should be floating-point, not integer.

2. the LOD parameter to sampling should be floating point as well. It's only integer for load(), just like the texture coordinate.

f32 texture_sample_compare(<depth_texture>, <comparison_sampler>, i32 | vec(2|3) coords, i32 bias)

3. I believe the bias mentioned (on the call?) is not the depth bias, it's the LOD bias. Perhaps, we need:

• texture_sample
• texture_sample_level (with f32 lod at the end)
• texture_sample_bias (with f32 bias at the end)

4. We probably don't want to allow sampling depth/comparison stuff between mipmap levels, so neither auto-lod, or floating-point lod, or bias, make sense. It should just be:

f32 texture_sample_compare(<depth_texture>, <comparison_sampler>, f32 | vec(2|3|4)<f32> coords, i32 lod)

[dneto0]

There's a lot going on here, but I wanted to answer one thing:

Does this only describe sample-able textures?

Yes, I believe so. In Vulkan SPIR-V an image type is either sampled (Sampled=1) or not-sampled (Sampled=2) (a storage image or storage texel buffer). To read a texel from a Sample=d=1 image, use OpImageFetch; to read a texel from a Sampled=2 image, use OpImageRead.

[kainino0x]

Experimentally, neither Metal nor Vulkan's validation layers complain if you depth-sample a color texture, though there's inconsistent behavior depending on the texture format / driver. Sometimes it follows the HLSL semantics and does the depth-comparison with the first component, sometimes it simply returns the value of the first component, and sometimes it returns something else.

Sounds like these are probably issues with the Vulkan and Metal validation layers, and we should ideally report them upstream.

[ben-clayton]

I've started a document that lists each of the texture functions by texture type for MSL and HLSL (SPIR-V TODO).

[ben-clayton]

Document updated with SPIR-V instructions

[Kangz]

Closing. Textures were added to WGSL with a bunch of functions. The only thing left seems to be the texture gather builtins, but I think it should be in a separate issue given how much was specced already.