[RFC] Query resolve

[msiglreith]

Overview

We have 3 types of queries in Vulkan: Occlusion, PipelineStatistics and Timestamp.
The first two are handled via vkCmdBeginQuery/vkCmdEndQuery and timestamps via vkCmdWriteTimestamp. Queries are allocated in heaps and marked as 'unavailable' at the beginning.
vkCmdBeginQuery calls activate queries and once the results can be obtained (after vkCmdEndQuery) the state will switch to available. vkResetQueryPool will reset them to unavailable.

Mapping to Metal

Metal doesn't expose any significant kind of timestamp or pipeline statistics support (afaik) and only supports Counting occlusion query.
Therefore, pipelineStatisticsQuery feature should be exposed as false, timestampComputeAndGraphics as false and timestampValidBits as 0 for all queues.
Occlusion queries are configured at renderpass boundary. Binary occlusion support is required and counting optional. Results are directly written to a buffer.

Mapping to D3D12

D3D12 uses query heaps, similar to Vulkan's query pools. The begin/end/write commands can be mapped without significant overhead to D3D12. Reset command implementation depends on the further Vulkan Portability API.
Resolving query data in D3D12 is implemented as a command list function ResolveQueryData(https://msdn.microsoft.com/de-de/library/windows/desktop/dn903896(v=vs.85).aspx). It write the current query values contiguously to a buffer (upload/download/local) as 64bit integers.
Vulkan exposes two functions to resolve queries: vkGetQueryPoolResults (device-side) and vkCmdCopyQueryPoolResults (command buffer-side). Beside additional effort to submit a command buffer with a query resolve to emulate the device function, the VkQueryResultFlags are difficult to handle:

• VK_QUERY_RESULT_64_BIT denotes that values are written as u64 values (default: u32)
• VK_QUERY_RESULT_WITH_AVAILABILITY_BIT adds an additional integer value indicating if the specific query is available to the buffer.
• VK_QUERY_RESULT_WAIT_BIT waits on all queries to be available, else resulting in a possible device loss.

Additionally, users can define which pipeline statistics can should be recorded and only marked statistics get written to the buffer. D3D12 will emit all statistics.

Issues

• Track the status (available/unavailable) of each query in the query heap to correctly emit the availability flags if requested. As these happen async it needs to be done in the command buffer. For example, compute shader to write availability values into an hidden buffer on command buffer end if we have Begin/EndQuery commands.
• Compute shader to handle the case where VK_QUERY_RESULT_64_BIT is not set.
• vkGetQueryPoolResults requires additional synchronization to ensure that queries have been processed by the command queues when VK_QUERY_RESULT_WAIT_BIT is set.

I think these are the main issues so far. Metal requires a bit further analysis but shouldn't cause major complications.

[kvark]

Note: this is currently a blocker for Dolphin on Metal. Adding the tag.