[BUG]: Freeze when starting AV capture during input replay or invalid codec

[phoenix1291]

Describe the Bug

📋 Environment

• PCSX2 Version: v2.5.26 (also tested with v2.3.410 for input recording)
• Operating System: Windows 10+
• CPU: Intel Core Ultra 7 266V
• GPU: Intel Arc 140V (driver 32.0.101.6913)
• Renderer: Vulkan
• Game Tested: Dead or Alive 2 – Hardcore (SCES-50003)
• Input Replay File: .p2m2 file created with PCSX2-v2.3.410

🧭 Summary
When launching AV capture (audio + video) during an active input replay session, PCSX2 either:

• Freezes entirely
• Returns codec initialization errors (especially with codecs like ffv1 or huffyuv)
• Fails to open the video encoder (avcodec_open2() returns error -22)
  Logs example:

GSCapture: preferred pixel format (23) was unsupported by the codec. Using (0) instead.
OSD [GSCaptureError]: avcodec_open2() for video failed: Invalid argument (-22)

🎯 Behavior

• AV capture works without replay mode active
• Replay mode works on its own
• But launching both simultaneously triggers failure
• Interestingly, if I retry AV capture one frame later, it works fine

💡 Hypothesis
This seems related to buffer initialization timing or thread desynchronization:

• Replay subsystem sets up input and graphics buffers
• GSCapture may be unable to access a valid pixel format or buffer until a few frames into replay
• Possibly a race condition between the capture thread and replay engine

✅ Temporary Workaround

• Start input replay
• Attempt AV capture
• If it fails, advance one frame (e.g. press F2)
• Retry capture — now works reliably

Reproduction Steps

When launching AV capture (audio + video) during an active input replay session, PCSX2 either:

• Freezes entirely
• Returns codec initialization errors (especially with codecs like ffv1 or huffyuv)
• Fails to open the video encoder (avcodec_open2() returns error -22)
  Logs example:

Expected Behavior

No response

PCSX2 Revision

PCSX2 Version: v2.5.26

Operating System

Windows 11

If Linux - Specify Distro

No response

Logs & Dumps

GSCapture: preferred pixel format (23) was unsupported by the codec. Using (0) instead.
OSD [GSCaptureError]: avcodec_open2() for video failed: Invalid argument (-22)