Attaching to running programs

As far as I know, attaching to and capturing from programs after they've started and begun rendering has only ever been done on consoles, however this is an extremely useful bit of functionality, and with a little bit of work it shouldn't be impossible to support on PC. It will almost certainly however require co-operation from the app to some greater or lesser degree.

Current theoretical plan:

1. Expose an API from RenderDoc to hooked applications to allow more flexible interop than currently exists.

2. Through this API we have two options which could be both implemented or we could pick one depending on what developers prefer:

3. The program voluntarily enters a lightweight capture mode at startup that has deliberately close to 0 overhead. No active capturing and no wrapping of resources/additional layer between program and API. On D3D11 for example this might only be used to store shader bytecode and input layout descriptors (two of the few things that cannot be queried after the fact).

   Then you essentially always run under this mode in development, without incurring any noticable overhead, and the program is in a 'capturable' state. RenderDoc can attach, upgrade from this lightweight mode into a more capable mode that includes the ability to capture and at this point query for any information, state and resource contents that exist.

   There will possibly be a requirement at this stage (API-dependent) that the application hand over its API objects to RenderDoc for possible wrapping or inspection, so the program would need to be able to iterate over all its API handles/pointers.

   This mode would have to be voluntary as the lightweight capture mode would probably be too brittle without the application being aware of it. Or conversely, I suspect that any capture mode that was robust enough to work on an unmodified application would be too heavyweight to leave permanently enabled even when you (initially) have no intention of capturing like this.

4. The program is aware of RenderDoc and can setup a callback/flagging/communication system that can be activated in the event that RenderDoc attaches to capture. Similar to above the application would have to be able to hand over its API objects, but it would also need to be able to hand over any information that RenderDoc can no longer obtain by coming in late - for example shader source/bytecode and such that cannot be retrieved from the real API.

---

This is all doable, but it would probably be a bunch of work. The integration into the target application is unfortunate but I don't think there's a way around it. In general regardless of the API there will be some requirement to modify pointers/handles that the application owns (even if we e.g. patched the vtables they point to or similar) and also there will be some required information to replay that cannot be obtained after the fact without co-operation with the application.