Limit max_binding_array_elements_per_shader_stage gets reset to 0, but why?

[John-Nagle]

I'm still struggling with WGPU 24 -> WGPU 30 conversion of Rend3.

Many tests fail with:

thread 'egui::test' (1129698) panicked at /home/john/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/wgpu-30.0.1/src/backend/wgpu_core.rs:1148:26:
wgpu error: Validation Error

Caused by:
  In Device::create_bind_group_layout, label = 'D2 texture bgl'
    Too many bindings of type BindingArrayElements in Stage ShaderStages(FRAGMENT), limit is 0, count was 131072. Check the limit `max_binding_array_elements_per_shader_stage` passed to `Adapter::request_device`

I've tried setting, in Limits,

max_binding_array_elements_per_shader_stage: 500000, /* new in WGPU 30*/

The device allocates and opens, but fails at bind group creation. In the error message, that limit is shown as 0.
Here's the allocated device and the features WGPU let me get:

Device acquired: Device { inner: Core(CoreDevice { context: ContextWgpuCore { type: "Native" }, id: DeviceId(0,1), error_sink: Mutex { data: ErrorSink }, features: Features { features_wgpu: FeaturesWGPU(TEXTURE_BINDING_ARRAY | BUFFER_BINDING_ARRAY | STORAGE_RESOURCE_BINDING_ARRAY | SAMPLED_TEXTURE_AND_STORAGE_BUFFER_ARRAY_NON_UNIFORM_INDEXING | PARTIALLY_BOUND_BINDING_ARRAY | MULTI_DRAW_INDIRECT_COUNT), features_webgpu: FeaturesWebGPU(DEPTH_CLIP_CONTROL | TEXTURE_COMPRESSION_BC | IMMEDIATES) } }) }
Adapter info: AdapterInfo { name: "NVIDIA GeForce RTX 3070", vendor: 4318, device: 9348, device_type: DiscreteGpu, device_pci_bus_id: "0000:09:00.0", driver: "NVIDIA", driver_info: "535.309.01", backend: Vulkan, subgroup_min_size: 32, subgroup_max_size: 32, transient_saves_memory: Some(false), limit_bucket: None }

Do I have to enable more features? But which one?

What seems to be going on is that, at WGPU 30, you only get the features you explicitly ask for when opening the adapter. Previously, in WGPU 24, you could ask for no features, and the adapter gave you everything it offered, which you could then check. There may be some additional feature I need to turn on for this to work. I tried requesting some more features:

        //  New for WGPU 30
        | Features::BUFFER_BINDING_ARRAY.bits()
        | Features::TEXTURE_BINDING_ARRAY.bits()
        | Features::STORAGE_RESOURCE_BINDING_ARRAY.bits()
        | Features::SAMPLED_TEXTURE_AND_STORAGE_BUFFER_ARRAY_NON_UNIFORM_INDEXING.bits()
        | Features::STORAGE_RESOURCE_BINDING_ARRAY.bits()

but it didn't help.

What am I missing here?

[andyleiserson]

Huh. The binding array limits can get zeroed out if InstanceFlags::STRICT_WEBGPU_COMPLIANCE is set, but that's not the default, and I thought that would affect device creation rather than bind group (layout) creation. Other than that I am not sure how this would be happening.

[John-Nagle]

It still has me puzzled, too. There are several things that force that limit to 0.
I'm going through WGPU trying to see what affects those limits.

Here's where that value gets set in WGPU:

wgpu/wgpu-hal/src/vulkan/adapter.rs

Line 1823 in dd033bf

max_binding_array_elements_per_shader_stage: max_binding_array_elements,

which gets its value from

https://github.com/gfx-rs/wgpu/blob/dd033bfb7fb3690eb2e80a971877a1ef8ecabe59/wgpu-hal/src/vulkan/adapter.rs#L1583C1-L1596C1

where the value is the min of rather a lot of values which seem to come from vk::PhysicalDeviceDescriptorIndexingPropertiesEXT.
That comes from the Vulkan driver for the device, I think. The device here is an NVidia 3070 on Ubuntu Linux, which ought to support most modern features. Driver is "535.309.01", which works for everything else but is a bit dated. Current driver is 595. Did WGPU 30 perhaps add a limit or feature not present in earlier drivers?

All this worked in WGPU 24 on the same hardware.

[John-Nagle]

I was hoping that this was some simple mistake I made. That would be easy to fix. Now this is starting to look like a WGPU bug that appears for some combinations of feature requests/backend/Vulkan driver/graphics card. What do you think?

If necessary, I can make up a branch of my test program so others can try it on other hardware. Should I do that?

[John-Nagle]

This still has me puzzled. The value for max_binding_array_elements_per_shader_stage is computed by WGPU as the minimum of these Vulkan values, which I got from "vulkaninfo" under Linux:

maxDescriptorSetUpdateAfterBindSampledImages         = 1048576
maxDescriptorSetUpdateAfterBindStorageImages         = 1048576
maxDescriptorSetUpdateAfterBindStorageBuffers        = 1048576
maxDescriptorSetUpdateAfterBindSampledImages         = 1048576
maxDescriptorSetUpdateAfterBindStorageImages         = 1048576
maxDescriptorSetUpdateAfterBindStorageBuffers        = 1048576


maxDescriptorSetUpdateAfterBindSamplers              = 1048576
maxPerStageDescriptorUpdateAfterBindSamplers         = 1048576

Those are only read if the Vulkan version is >= 1.2. It's 1.3. So that should work.

Haven't yet figured out what's setting max_binding_array_elements_per_shader_stage to zero. I don't see anything the caller to WGPU does which can affect this. It's finding the right graphics device.

[John-Nagle]

So I dumped the Limits obtained from the adapter:

Device acquired: Device { inner: Core(CoreDevice { context: ContextWgpuCore { type: "Native" }, id: DeviceId(0,1), error_sink: Mutex { data: ErrorSink }, features: Features { features_wgpu: FeaturesWGPU(TIMESTAMP_QUERY_INSIDE_PASSES | TEXTURE_BINDING_ARRAY | BUFFER_BINDING_ARRAY | STORAGE_RESOURCE_BINDING_ARRAY | SAMPLED_TEXTURE_AND_STORAGE_BUFFER_ARRAY_NON_UNIFORM_INDEXING | PARTIALLY_BOUND_BINDING_ARRAY | MULTI_DRAW_INDIRECT_COUNT), features_webgpu: FeaturesWebGPU(DEPTH_CLIP_CONTROL | TEXTURE_COMPRESSION_BC | TIMESTAMP_QUERY | IMMEDIATES) } }) }
Adapter info: AdapterInfo { name: "NVIDIA GeForce RTX 3070", vendor: 4318, device: 9348, device_type: DiscreteGpu, device_pci_bus_id: "0000:09:00.0", driver: "NVIDIA", driver_info: "535.309.01", backend: Vulkan, subgroup_min_size: 32, subgroup_max_size: 32, transient_saves_memory: Some(false), limit_bucket: None }
Limits: Limits { max_texture_dimension_1d: 32768, 
max_texture_dimension_2d: 32768, 
max_texture_dimension_3d: 16384, 
max_texture_array_layers: 2048, 
max_bind_groups: 8, 
max_bind_groups_plus_vertex_buffers: 24, 
max_bindings_per_bind_group: 4294967295, 
max_dynamic_uniform_buffers_per_pipeline_layout: 15, 
max_dynamic_storage_buffers_per_pipeline_layout: 16, 
max_sampled_textures_per_shader_stage: 524288, 
max_samplers_per_shader_stage: 524288, 
max_storage_buffers_per_shader_stage: 524288, 
max_storage_textures_per_shader_stage: 524288, 
max_uniform_buffers_per_shader_stage: 524288, 
max_binding_array_elements_per_shader_stage: 0, 
max_binding_array_acceleration_structure_elements_per_shader_stage: 0,
max_binding_array_sampler_elements_per_shader_stage: 0, 
max_uniform_buffer_binding_size: 65536, 
max_storage_buffer_binding_size: 2147483644, 
max_vertex_buffers: 16, 
max_buffer_size: 4292870144, 
max_vertex_attributes: 32, 
max_vertex_buffer_array_stride: 2048, 
max_inter_stage_shader_variables: 31, 
min_uniform_buffer_offset_alignment: 64, 
min_storage_buffer_offset_alignment: 32, 
max_color_attachments: 8, 
max_color_attachment_bytes_per_sample: 128, 
max_compute_workgroup_storage_size: 49152, 
max_compute_invocations_per_workgroup: 1024, 
max_compute_workgroup_size_x: 1024, 
max_compute_workgroup_size_y: 1024, 
max_compute_workgroup_size_z: 64, 
max_compute_workgroups_per_dimension: 65535, 
max_immediate_size: 256, 
max_non_sampler_bindings: 1000000, 
max_task_workgroup_total_count: 0, 
max_task_workgroups_per_dimension: 0, 
max_mesh_workgroup_total_count: 0, 
max_mesh_workgroups_per_dimension: 0, 
max_task_invocations_per_workgroup: 0, 
max_task_invocations_per_dimension: 0, 
max_mesh_invocations_per_workgroup: 0, 
max_mesh_invocations_per_dimension: 0, 
max_task_payload_size: 0, 
max_mesh_output_vertices: 0, 
max_mesh_output_primitives: 0, 
max_mesh_output_layers: 0, max_mesh_multiview_view_count: 0, 
max_blas_primitive_count: 0, max_blas_geometry_count: 0, 
max_tlas_instance_count: 0, 
max_acceleration_structures_per_shader_stage: 0, 
max_buffers_and_acceleration_structures_per_shader_stage: 28, 
max_multiview_view_count: 0, 
max_ray_dispatch_count: 0, 
max_ray_recursion_depth: 0 }

thread 'textured_quad::test' (2132248) panicked at /home/john/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/wgpu-30.0.1/src/backend/wgpu_core.rs:1148:26:
wgpu error: Validation Error

Caused by:
  In Device::create_bind_group_layout, label = 'D2 texture bgl'
    Too many bindings of type BindingArrayElements in Stage ShaderStages(FRAGMENT), limit is 0, count was 131072. Check the limit `max_binding_array_elements_per_shader_stage` passed to `Adapter::request_device`

So that's what WGPU is saying that Vulkan reported. Mesh shaders and ray tracing features were not requested, which explains all those zero entries at the end. That's fine. Trouble comes from

max_binding_array_elements_per_shader_stage: 0, 
max_binding_array_acceleration_structure_elements_per_shader_stage: 0,
max_binding_array_sampler_elements_per_shader_stage: 0, 

which are values WGPU calculates, as mentioned above. Those should not be 0. It's as if binding arrays were turned off, for which WGPU zeroes out those values. But features
TEXTURE_BINDING_ARRAY | BUFFER_BINDING_ARRAY | STORAGE_RESOURCE_BINDING_ARRAY are all on. So this should not be happening.

We've seen from the vulkaninfo data above that Vulkan has reasonable numbers for the Vk fields used to compute those WGPU values.

This looks more and more like a WGPU bug. Will look at adapter.rs some more.

[John-Nagle]

More dumping. Built WGPU 30.0.1 from Github source so I could add println output in adapter.rs:

Added at line 1533:
println!("Descriptor indexing info: {:?}\nMax binding array elements: {}", self.descriptor_indexing, max_binding_array_elements); // ***TEMP DEBUG***

Descriptor indexing info: Some(PhysicalDeviceDescriptorIndexingProperties { s_type: PHYSICAL_DEVICE_DESCRIPTOR_INDEXING_PROPERTIES, p_next: 0x72dc04feefb8, max_update_after_bind_descriptors_in_all_pools: 4294967295, shader_uniform_buffer_array_non_uniform_indexing_native: 1, shader_sampled_image_array_non_uniform_indexing_native: 1, shader_storage_buffer_array_non_uniform_indexing_native: 1, shader_storage_image_array_non_uniform_indexing_native: 1, shader_input_attachment_array_non_uniform_indexing_native: 1, robust_buffer_access_update_after_bind: 1, quad_divergent_implicit_lod: 1, max_per_stage_descriptor_update_after_bind_samplers: 1048576, max_per_stage_descriptor_update_after_bind_uniform_buffers: 1048576, max_per_stage_descriptor_update_after_bind_storage_buffers: 1048576, max_per_stage_descriptor_update_after_bind_sampled_images: 1048576, max_per_stage_descriptor_update_after_bind_storage_images: 1048576, max_per_stage_descriptor_update_after_bind_input_attachments: 1048576, max_per_stage_update_after_bind_resources: 4294967295, max_descriptor_set_update_after_bind_samplers: 1048576, max_descriptor_set_update_after_bind_uniform_buffers: 1048576, max_descriptor_set_update_after_bind_uniform_buffers_dynamic: 15, max_descriptor_set_update_after_bind_storage_buffers: 1048576, max_descriptor_set_update_after_bind_storage_buffers_dynamic: 16, max_descriptor_set_update_after_bind_sampled_images: 1048576, max_descriptor_set_update_after_bind_storage_images: 1048576, max_descriptor_set_update_after_bind_input_attachments: 1048576, _marker: PhantomData<&()> })

Max binding array elements: 1048576

So everything is on track at that point. This is before adjust_raw_limits is called.
After adjust_raw_limits, values are still good. No unexpected zeroes:

Limits after adjust_raw_limits: Limits { max_texture_dimension_1d: 32768, max_texture_dimension_2d: 32768, max_texture_dimension_3d: 16384, max_texture_array_layers: 2048, max_bind_groups: 8, max_bind_groups_plus_vertex_buffers: 24, max_bindings_per_bind_group: 4294967295, max_dynamic_uniform_buffers_per_pipeline_layout: 15, max_dynamic_storage_buffers_per_pipeline_layout: 16, max_sampled_textures_per_shader_stage: 524288, max_samplers_per_shader_stage: 524288, max_storage_buffers_per_shader_stage: 524288, max_storage_textures_per_shader_stage: 524288, max_uniform_buffers_per_shader_stage: 524288, **max_binding_array_elements_per_shader_stage: 1048576**, max_binding_array_acceleration_structure_elements_per_shader_stage: 524288, max_binding_array_sampler_elements_per_shader_stage: 1048576, max_uniform_buffer_binding_size: 65536, max_storage_buffer_binding_size: 2147483644, max_vertex_buffers: 16, max_buffer_size: 4292870144, max_vertex_attributes: 32, max_vertex_buffer_array_stride: 2048, max_inter_stage_shader_variables: 31, min_uniform_buffer_offset_alignment: 64, min_storage_buffer_offset_alignment: 32, max_color_attachments: 8, max_color_attachment_bytes_per_sample: 128, max_compute_workgroup_storage_size: 49152, max_compute_invocations_per_workgroup: 1024, max_compute_workgroup_size_x: 1024, max_compute_workgroup_size_y: 1024, max_compute_workgroup_size_z: 64, max_compute_workgroups_per_dimension: 65535, max_immediate_size: 256, max_non_sampler_bindings: 4294967295, max_task_workgroup_total_count: 4194304, max_task_workgroups_per_dimension: 65535, max_mesh_workgroup_total_count: 4194304, max_mesh_workgroups_per_dimension: 65535, max_task_invocations_per_workgroup: 128, max_task_invocations_per_dimension: 128, max_mesh_invocations_per_workgroup: 128, max_mesh_invocations_per_dimension: 128, max_task_payload_size: 16384, max_mesh_output_vertices: 256, max_mesh_output_primitives: 256, max_mesh_output_layers: 2048, max_mesh_multiview_view_count: 4, max_blas_primitive_count: 536870911, max_blas_geometry_count: 16777215, max_tlas_instance_count: 16777215, max_acceleration_structures_per_shader_stage: 524288, max_buffers_and_acceleration_structures_per_shader_stage: 4294967295, max_multiview_view_count: 32, max_ray_dispatch_count: 1073741824, max_ray_recursion_depth: 31 }
Things are fine at this point. Who messes with Limits later?

[John-Nagle]

Onward: Captured Limits at create bind group. Now they're broken:

Create bind group layout limits: Limits { max_texture_dimension_1d: 8192, max_texture_dimension_2d: 8192, max_texture_dimension_3d: 2048, max_texture_array_layers: 256, max_bind_groups: 4, max_bind_groups_plus_vertex_buffers: 24, max_bindings_per_bind_group: 1000, max_dynamic_uniform_buffers_per_pipeline_layout: 8, max_dynamic_storage_buffers_per_pipeline_layout: 4, max_sampled_textures_per_shader_stage: 16, max_samplers_per_shader_stage: 16, max_storage_buffers_per_shader_stage: 8, max_storage_textures_per_shader_stage: 4, max_uniform_buffers_per_shader_stage: 12, max_binding_array_elements_per_shader_stage: 0, max_binding_array_acceleration_structure_elements_per_shader_stage: 0, max_binding_array_sampler_elements_per_shader_stage: 0, max_uniform_buffer_binding_size: 65536, max_storage_buffer_binding_size: 134217728, max_vertex_buffers: 8, max_buffer_size: 268435456, max_vertex_attributes: 16, max_vertex_buffer_array_stride: 2048, max_inter_stage_shader_variables: 16, min_uniform_buffer_offset_alignment: 256, min_storage_buffer_offset_alignment: 256, max_color_attachments: 8, max_color_attachment_bytes_per_sample: 32, max_compute_workgroup_storage_size: 16384, max_compute_invocations_per_workgroup: 256, max_compute_workgroup_size_x: 256, max_compute_workgroup_size_y: 256, max_compute_workgroup_size_z: 64, max_compute_workgroups_per_dimension: 65535, max_immediate_size: 0, max_non_sampler_bindings: 1000000, max_task_workgroup_total_count: 0, max_task_workgroups_per_dimension: 0, max_mesh_workgroup_total_count: 0, max_mesh_workgroups_per_dimension: 0, max_task_invocations_per_workgroup: 0, max_task_invocations_per_dimension: 0, max_mesh_invocations_per_workgroup: 0, max_mesh_invocations_per_dimension: 0, max_task_payload_size: 0, max_mesh_output_vertices: 0, max_mesh_output_primitives: 0, max_mesh_output_layers: 0, max_mesh_multiview_view_count: 0, max_blas_primitive_count: 0, max_blas_geometry_count: 0, max_tlas_instance_count: 0, max_acceleration_structures_per_shader_stage: 0, max_buffers_and_acceleration_structures_per_shader_stage: 28, max_multiview_view_count: 0, max_ray_dispatch_count: 0, max_ray_recursion_depth: 0 }

Note

max_binding_array_elements_per_shader_stage: 0,

Somewhere between device setup and the first bind, something zeroed that out. But where is that happening?

[John-Nagle]

New info: the Adapter and the Device both have Limits, and they may not be the same. More later.