False Negative Validation Error: VUID-vkCmdWriteTimestamp-None-00830

[tteguh]

Environment:

• OS: Windows 10
• GPU and driver version: RTX 3090 528.81
• SDK or header version if building from repo: top of tree (204ba5ae8)
• Options enabled (synchronization, best practices, etc.):

Describe the Issue

The validation layer don't seem to properly respect the implicit ordering guarantees for timestamp query operations that should be guaranteed based on the vulkan spec: https://registry.khronos.org/vulkan/specs/1.3-extensions/html/vkspec.html#synchronization-implicit

"3. The order in which command buffers are specified in the pCommandBuffers member of VkSubmitInfo or VkSubmitInfo2 from lowest index to highest."

The example code attached describes the scenario that is triggering the false negative:

Validation Error: [ VUID-vkCmdWriteTimestamp-None-00830 ] Object 0: handle = 0x26e709d9a00, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0xf56c9b0000000004, type = VK_OBJECT_TYPE_QUERY_POOL; | MessageID = 0xeb0b9b05 | vkCmdWriteTimestamp(): VkQueryPool 0xf56c9b0000000004[] and query 0: query not reset. After query pool creation, each query must be reset before it is used. Queries must also be reset between uses. The Vulkan spec states: All queries used by the command must be unavailable (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-vkCmdWriteTimestamp-None-00830)

The vkCmdTimestampWrite and vkCmdResetQueryPool happens in two different command buffers. Submitting this into the same VkSubmitInfo in correct order from lowest index to highest will trigger the validation error.

Splitting it such that the command buffers are submitted within it's own VkSubmitInfo fixes the problem. Set splitSubmit in the example code to true to see this behavior.

Expected behavior

The validation error should not be present.

Valid Usage ID
VUID-vkCmdWriteTimestamp-None-00830

Additional context

Test code that demonstrates false negative validation error

TEST_F(PositiveQuery, MyTest) {

    RETURN_IF_SKIP(Init());

    vkt::QueryPool query_pool(*m_device, VK_QUERY_TYPE_TIMESTAMP, 2);

    vkt::Buffer buffer(*m_device, 4 * sizeof(uint64_t), VK_BUFFER_USAGE_TRANSFER_DST_BIT);

    VkCommandBuffer command_buffer[3];
    VkCommandBufferAllocateInfo command_buffer_allocate_info = vku::InitStructHelper();
    command_buffer_allocate_info.commandPool = m_command_pool.handle();
    command_buffer_allocate_info.commandBufferCount = 3;
    command_buffer_allocate_info.level = VK_COMMAND_BUFFER_LEVEL_PRIMARY;
    vk::AllocateCommandBuffers(device(), &command_buffer_allocate_info, command_buffer);

    {
        VkCommandBufferBeginInfo begin_info = vku::InitStructHelper();

        // First command buffer to reset the query pool.  Only submitted once
        vk::BeginCommandBuffer(command_buffer[0], &begin_info);
        vk::CmdResetQueryPool(command_buffer[0], query_pool.handle(), 0, 2);
        vk::EndCommandBuffer(command_buffer[0]);

        vk::BeginCommandBuffer(command_buffer[1], &begin_info);
        vk::CmdWriteTimestamp(command_buffer[1], VK_PIPELINE_STAGE_BOTTOM_OF_PIPE_BIT, query_pool.handle(), 0);
        vk::EndCommandBuffer(command_buffer[1]);

        vk::BeginCommandBuffer(command_buffer[2], &begin_info);
        vk::CmdWriteTimestamp(command_buffer[2], VK_PIPELINE_STAGE_BOTTOM_OF_PIPE_BIT, query_pool.handle(), 1);
        vk::CmdCopyQueryPoolResults(command_buffer[2], query_pool.handle(), 0, 2, buffer.handle(), 0, sizeof(uint64_t), VK_QUERY_RESULT_64_BIT | VK_QUERY_RESULT_WAIT_BIT);
        vk::CmdResetQueryPool(command_buffer[2], query_pool.handle(), 0, 2);
        vk::EndCommandBuffer(command_buffer[2]);
    }

    const uint32_t kLoopCount = 2;
    const bool splitSubmit = false; // True = No validation error.  False = validation error
    for (uint32_t loop = 0; loop < kLoopCount; ++loop)
    {
        // Splitting the command buffer into different submit infos seem to solve the problem
        if (splitSubmit)
        {
            VkSubmitInfo submit_info[3] {};

            submit_info[0] = vku::InitStructHelper();
            submit_info[0].commandBufferCount = 1;
            submit_info[0].pCommandBuffers = &command_buffer[0];
            submit_info[0].signalSemaphoreCount = 0;
            submit_info[0].pSignalSemaphores = nullptr;

            submit_info[1] = vku::InitStructHelper();
            submit_info[1].commandBufferCount = 1;
            submit_info[1].pCommandBuffers = &command_buffer[1];
            submit_info[1].signalSemaphoreCount = 0;
            submit_info[1].pSignalSemaphores = nullptr;

            submit_info[2] = vku::InitStructHelper();
            submit_info[2].commandBufferCount = 1;
            submit_info[2].pCommandBuffers = &command_buffer[2];
            submit_info[2].signalSemaphoreCount = 0;
            submit_info[2].pSignalSemaphores = nullptr;

            if (loop == 0)
            {
                vk::QueueSubmit(m_default_queue->handle(), 3, &submit_info[0], VK_NULL_HANDLE);
            }
            else
            {
                vk::QueueSubmit(m_default_queue->handle(), 2, &submit_info[1], VK_NULL_HANDLE);
            }
        }
        else
        {
            VkSubmitInfo submit_info{};
            submit_info = vku::InitStructHelper();
            submit_info.commandBufferCount = loop == 0 ? 3 : 2;
            submit_info.pCommandBuffers = loop == 0 ? &command_buffer[0] : &command_buffer[1];
            submit_info.signalSemaphoreCount = 0;
            submit_info.pSignalSemaphores = nullptr;

            vk::QueueSubmit(m_default_queue->handle(), 1, &submit_info, VK_NULL_HANDLE);
        }

        m_default_queue->Wait();
    }

    vk::FreeCommandBuffers(device(), m_command_pool.handle(), 2, command_buffer);
}

[spencer-lunarg]

UNASSIGNED-CoreValidation-DrawState-QueryNotReset

this is not the VVL anymore (I remember taking it out a while ago)

SDK or header version if building from repo: 1.3.289

you are building the VVL from the latest top of tree, or are you using and SDK built version (if so, which version?)

[tteguh]

Sorry, turns out I had environment vars that loaded the old vulkan validation layer instead of the one from the repo.

I've updated the issue with the correct validation error message and version. The issue is still relevant, although we do get a different validation id with a similar message. Test was ran with the layer from latest top of tree

[spencer-lunarg]

The issue is still relevant, although we do get a different validation id with a similar message.

It would be very helpful to know what the error message looks like now

[tteguh]

It would be very helpful to know what the error message looks like now

The issue has been modified with the new validation message already along with my previous comment

Validation Error: [ VUID-vkCmdWriteTimestamp-None-00830 ] Object 0: handle = 0x26e709d9a00, type = VK_OBJECT_TYPE_COMMAND_BUFFER; Object 1: handle = 0xf56c9b0000000004, type = VK_OBJECT_TYPE_QUERY_POOL; | MessageID = 0xeb0b9b05 | vkCmdWriteTimestamp(): VkQueryPool 0xf56c9b0000000004[] and query 0: query not reset. After query pool creation, each query must be reset before it is used. Queries must also be reset between uses. The Vulkan spec states: All queries used by the command must be unavailable (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/html/vkspec.html#VUID-vkCmdWriteTimestamp-None-00830)

[spencer-lunarg]

Thanks @tteguh ! I will take a look at it early next week and get it fixed!

[spencer-lunarg]

Ok, so confirmed the issue is around

    // The "local" prefix is about tracking state within a *single* queue submission
    // (across all command buffers of that submission), as opposed to globally
    // tracking state accross *all* submissions to the same queue.
    QueryMap local_query_to_state_map;

which is why it doesn't catch the vkCmdResetQueryPool at the end of the previous command buffer... will need time to really digest the query logic to fix this, which unfortunately might be a while 😞

[tteguh]

No worries @spencer-lunarg. Thanks for the update!