🐛 Fix recording stopping when user triggers UAC elevation

.github/copilot-instructions.md

@@ -3,7 +3,9 @@
 ## Code Style & Formatting Standards
 
 ### Commit Messages (Conventional Commits)
+
 Use the following prefixes for all commits:
+
 - `feat:` New features
 - `fix:` Bug fixes
 - `docs:` Documentation changes
@@ -18,7 +20,9 @@ Use the following prefixes for all commits:
 Example: `feat: Add ETA automation with timezone awareness`
 
 ### PR Titles (Emoji Prefix)
+
 Use emoji prefix followed by brief description:
+
 - `✨ Add new feature`
 - `🐛 Fix bug or issue`
 - `📚 Update documentation`
@@ -31,25 +35,33 @@ Use emoji prefix followed by brief description:
 Example: `✨ Add interactive task selection with rich UI`
 
 ### PR Body (GitHub-Flavored Markdown)
+
 Structure all PR descriptions with these sections:
+
 ```markdown
 ### What does this PR do?
+
 Brief explanation of changes and what was implemented.
 
 ### Why are we doing this?
+
 Context, motivation, and reason for the changes.
 
 ### How should this be tested?
+
 Testing instructions, test cases, and validation steps.
 
 ### Any deployment notes?
+
 Environment variables, migrations, breaking changes, or special instructions.
 ```
 
 Include related issue references: `Closes #71, #77` (at end of description)
 
 ### PR Metadata Requirements
+
 Always ensure the following metadata is set on every PR:
+
 - **Labels**: Assign relevant labels (e.g., `enhancement`, `bug`, `documentation`, `refactor`, `testing`)
 - **Assignees**: Assign to yourself (J-MaFf)
 - **Issues**: Link all related issues in the PR description and GitHub's linked issues feature
@@ -75,24 +87,56 @@ Note: Use `gh issue edit` for both issues and pull requests. Replace `<PR_NUMBER
 ## Project-Specific Conventions
 
 ### SFA Certificate Management Scripts
+
 The `/Scripts/SFA/` directory contains the integrated certificate distribution system:
 
 **Workflow:**
+
 1. **Export-UserCertificates.ps1** - Extracts certificates from Windows Certificate Store
 2. **Publish-SFACertificates.ps1** - Distributes certificates to 24+ branch servers with pre-flight connectivity checks
 3. **Move-ExpiredUserCertificates.ps1** - Archives expired certificates automatically
 
 **Key Features:**
+
 - Pre-flight connectivity check integrated into Publish-SFACertificates.ps1
 - User confirmation prompts for partially accessible branches
 - Deduplication: skips certificates already present on remote servers
 - Automatic cleanup of expired certificates
 - Timestamped reports and CSV exports
 
 **When Making Changes:**
+
 - Update both script help documentation and requirements.md file
 - Test connectivity handling and user prompts
 - Verify report generation accuracy
 - Update README.html to reflect workflow changes
 - Use `refactor:` commits when integrating diagnostic tools (e.g., Test-BranchMappings)
 - Use `feat:` commits for new connectivity/workflow improvements
+
+### iKAT Automated Screen Recording
+
+The `/Scripts/iKAT/Invoke-FFmpegCapture/` directory contains the session recording system deployed to `KFWS6IKAT01.kikkoman.com`.
+
+**Purpose:** Capture `dpuerner`'s full RDS desktop sessions to diagnose intermittent iKAT application errors.
+
+**Workflow:**
+
+1. **Invoke-iKATRecording.ps1** — Core script: validates user, checks disk space, launches FFmpeg in a restart loop
+2. **Start-DinaRecording.ps1** / **Start-JoeyRecording.ps1** — Runner scripts called by Task Scheduler via `.vbs` launchers
+3. **Remove-ExpiredRecordings.ps1** — Daily cleanup of `.ts` files older than the retention period
+
+**Critical Constraints — do not change without understanding the impact:**
+
+- **Output format must stay MPEG-TS (`.ts`)** — MP4/MKV require a trailer; if FFmpeg is force-killed at logoff, the file is unplayable. MPEG-TS packets are self-contained with no trailer needed.
+- **`-tune zerolatency` must stay** — disables libx264 lookahead buffer; without it, up to 5 seconds of encoded frames are held in memory and lost on abrupt kill.
+- **`-flush_packets 1` must stay** — forces OS-level flush after every TS packet; removing it re-introduces data loss at session end.
+- **Task Scheduler tasks must use `LogonType Interactive` + `RunLevel Limited`** — `gdigrab` cannot access the RDS desktop from Session 0 or when running elevated.
+- **The restart loop** (`while ($true)` with exit-code check) handles UAC secure desktop transitions that force-kill FFmpeg mid-session. Do not replace with a single-launch pattern.
+- **Session end must be detected via `explorer.exe`**, not FFmpeg's exit code — UAC secure desktop transitions cause gdigrab to exit with code `0` (clean), not a crash code. Checking exit code `0` to break the loop would stop recording at every UAC prompt.
+
+**When Making Changes:**
+
+- Always update `log.md` with a dated entry describing what changed and why
+- The restart guardrails (`$maxRestartCount`, `$restartWindowMinutes`) protect against infinite crash loops — adjust values carefully
+- `Remove-ExpiredRecordings.ps1` filter must match the output extension (currently `*.ts`) — keep in sync if format ever changes
+- Test with both a clean logoff (graceful) and an abrupt logoff (hard disconnect) to verify no data loss in either case

Scripts/iKAT/Invoke-FFmpegCapture/Invoke-iKATRecording.ps1

@@ -18,6 +18,7 @@ if (-not (Test-Path $logDir)) { New-Item -ItemType Directory -Path $logDir -Forc
 "$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss') - Script started. USERNAME=$($env:USERNAME), TargetUser=$TargetUser" | Add-Content "$logDir\debug.log"
 
 if ($env:USERNAME -ne $TargetUser) {
+    Write-Output "Current user '$($env:USERNAME)' does not match target user '$TargetUser'. Exiting."
     "$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss') - User mismatch. Exiting." | Add-Content "$logDir\debug.log"
     Exit
 }
@@ -35,6 +36,7 @@ if ($drive) {
     $freeSpaceGB = [math]::Round($drive.FreeSpace / 1GB, 2)
     "$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss') - Free space: $freeSpaceGB GB, Required: $MinFreeSpaceGB GB" | Add-Content "$logDir\debug.log"
     if ($freeSpaceGB -lt $MinFreeSpaceGB) {
+        Write-Warning "Not enough free space. $freeSpaceGB GB available, $MinFreeSpaceGB GB required."
         "$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss') - Not enough free space. Exiting." | Add-Content "$logDir\debug.log"
         Exit
     }
@@ -46,50 +48,105 @@ Add-Type -AssemblyName System.Windows.Forms
 $virtualScreen = [System.Windows.Forms.SystemInformation]::VirtualScreen
 "$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss') - Reported VirtualScreen: $($virtualScreen.Width)x$($virtualScreen.Height) offset $($virtualScreen.X),$($virtualScreen.Y)" | Add-Content "$logDir\debug.log"
 
-# 4. Prepare output file
-$timestamp  = Get-Date -Format "yyyyMMdd_HHmmss"
-$outputFile = Join-Path -Path $OutputDir -ChildPath "$($TargetUser)_session_$timestamp.ts"
-
-# Construct the FFmpeg arguments
-# - gdigrab with no -video_size/-offset: auto-detects the full desktop at true physical pixel dimensions,
-#   avoiding DPI scaling mismatches that caused the resolution to appear cropped on RDS sessions.
-# - scale filter: rounds width and height down to even numbers required by libx264 (no resize, just crop 1px if odd).
-# - tune zerolatency: disables libx264 lookahead and B-frames so each frame is encoded and released immediately.
-# - g 5: 1-second keyframe interval (at 5fps); combined with zerolatency the encoder buffer stays under 1 second.
-# - MPEG-TS: self-contained 188-byte packets, fully playable with no trailer needed.
-# - flush_packets 1: flush every TS packet to disk immediately after encoding.
-$ffmpegArgsStr = "-f gdigrab -framerate 5 -i desktop -c:v libx264 -preset ultrafast -tune zerolatency -crf 30 -pix_fmt yuv420p -g 5 -vf `"scale=trunc(iw/2)*2:trunc(ih/2)*2`" -flush_packets 1 -f mpegts `"$outputFile`""
-
-# 5. Start FFmpeg silently but intercept StandardInput for graceful shutdown
-$procInfo = New-Object System.Diagnostics.ProcessStartInfo
-$procInfo.FileName = $FFmpegPath
-$procInfo.Arguments = $ffmpegArgsStr
-$procInfo.RedirectStandardInput = $true
-$procInfo.UseShellExecute = $false
-$procInfo.CreateNoWindow = $true
-
-"$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss') - Launching ffmpeg: $ffmpegArgsStr" | Add-Content "$logDir\debug.log"
-$ffmpegProcess = [System.Diagnostics.Process]::Start($procInfo)
-"$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss') - FFmpeg PID: $($ffmpegProcess.Id)" | Add-Content "$logDir\debug.log"
-
-try {
-    # 6. Block until FFmpeg exits naturally or the script is interrupted (logoff/session end)
-    while (-not $ffmpegProcess.HasExited) {
-        Start-Sleep -Seconds 5
+# 4. Restart guardrails for unexpected FFmpeg exits (ex: UAC secure desktop transitions)
+# - First 10 restarts: 1-second delay (handles rapid UAC transitions without noticeable lag)
+# - Restarts 11+: logarithmic backoff so a genuinely broken FFmpeg backs off gradually.
+#   Formula: Round(Ln(restartCount - 9) * 5) seconds
+#   e.g. restart 11=3s, restart 15=9s, restart 20=12s, restart 30=15s, restart 60=20s
+$restartWindowMinutes = 60
+$maxRestartCount = 60
+$restartCount = 0
+$restartWindowStart = Get-Date
+
+# Session timestamp is set once per logon — shared across all segment files for this session.
+$sessionTimestamp = Get-Date -Format "yyyyMMdd_HHmmss"
+$partNumber = 1
+
+while ($true) {
+    # 4. Prepare output file — each segment shares the session timestamp and gets an incrementing part number.
+    $outputFile = Join-Path -Path $OutputDir -ChildPath "$($TargetUser)_${sessionTimestamp}_part${partNumber}.ts"
+
+    # Construct the FFmpeg arguments
+    # - gdigrab with no -video_size/-offset: auto-detects the full desktop at true physical pixel dimensions,
+    #   avoiding DPI scaling mismatches that caused the resolution to appear cropped on RDS sessions.
+    # - scale filter: rounds width and height down to even numbers required by libx264 (no resize, just crop 1px if odd).
+    # - tune zerolatency: disables libx264 lookahead and B-frames so each frame is encoded and released immediately.
+    # - g 5: 1-second keyframe interval (at 5fps); combined with zerolatency the encoder buffer stays under 1 second.
+    # - MPEG-TS: self-contained 188-byte packets, fully playable with no trailer needed.
+    # - flush_packets 1: flush every TS packet to disk immediately after encoding.
+    $ffmpegArgsStr = "-f gdigrab -framerate 5 -i desktop -c:v libx264 -preset ultrafast -tune zerolatency -crf 30 -pix_fmt yuv420p -g 5 -vf `"scale=trunc(iw/2)*2:trunc(ih/2)*2`" -flush_packets 1 -f mpegts `"$outputFile`""
+
+    # 5. Start FFmpeg silently but intercept StandardInput for graceful shutdown
+    $procInfo = New-Object System.Diagnostics.ProcessStartInfo
+    $procInfo.FileName = $FFmpegPath
+    $procInfo.Arguments = $ffmpegArgsStr
+    $procInfo.RedirectStandardInput = $true
+    $procInfo.UseShellExecute = $false
+    $procInfo.CreateNoWindow = $true
+
+    "$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss') - Launching ffmpeg: $ffmpegArgsStr" | Add-Content "$logDir\debug.log"
+    try {
+        $ffmpegProcess = [System.Diagnostics.Process]::Start($procInfo)
     }
-}
-finally {
-    # 7. Session ended or script was interrupted — terminate FFmpeg gracefully
-    if ($null -ne $ffmpegProcess -and -not $ffmpegProcess.HasExited) {
-        # Send 'q' to gracefully stop recording so the file header formatting writes correctly
-        $ffmpegProcess.StandardInput.WriteLine("q")
-
-        # Wait up to 10 seconds for it to write headers and close
-        $ffmpegProcess.WaitForExit(10000) | Out-Null
-
-        # Fallback if the process stubbornly hung
-        if (-not $ffmpegProcess.HasExited) {
-            $ffmpegProcess.Kill()
+    catch {
+        "$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss') - Failed to start ffmpeg: $($_.Exception.Message)" | Add-Content "$logDir\debug.log"
+        throw
+    }
+    "$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss') - FFmpeg PID: $($ffmpegProcess.Id)" | Add-Content "$logDir\debug.log"
+
+    try {
+        # 6. Block until FFmpeg exits naturally or the script is interrupted (logoff/session end)
+        while (-not $ffmpegProcess.HasExited) {
+            Start-Sleep -Seconds 5
+        }
+    }
+    finally {
+        # 7. Session ended or script was interrupted — terminate FFmpeg gracefully
+        if ($null -ne $ffmpegProcess -and -not $ffmpegProcess.HasExited) {
+            # Send 'q' to gracefully stop recording so the file header formatting writes correctly
+            $ffmpegProcess.StandardInput.WriteLine("q")
+
+            # Wait up to 10 seconds for it to write headers and close
+            $ffmpegProcess.WaitForExit(10000) | Out-Null
+
+            # Fallback if the process stubbornly hung
+            if (-not $ffmpegProcess.HasExited) {
+                $ffmpegProcess.Kill()
+            }
         }
     }
-}
+
+    $exitCode = $ffmpegProcess.ExitCode
+    "$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss') - FFmpeg exited with code $exitCode" | Add-Content "$logDir\debug.log"
+
+    # Detect session end via explorer.exe — it only runs in an active interactive session.
+    # If explorer is gone, the user has logged off (regardless of FFmpeg's exit code).
+    # If explorer is still running, FFmpeg was killed mid-session (e.g., UAC secure desktop
+    # transition closes the gdigrab handle cleanly with exit code 0) — restart the recording.
+    if (-not (Get-Process -Name "explorer" -ErrorAction SilentlyContinue)) {
+        "$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss') - Explorer not found; user has logged off. Stopping." | Add-Content "$logDir\debug.log"
+        break
+    }
+
+    $partNumber++
+    "$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss') - Explorer still running; FFmpeg exited mid-session. Starting segment $partNumber." | Add-Content "$logDir\debug.log"
+
+    $now = Get-Date
+    if ($now -gt $restartWindowStart.AddMinutes($restartWindowMinutes)) {
+        $restartCount = 0
+        $restartWindowStart = $now
+    }
+
+    $restartCount += 1
+
+    if ($restartCount -gt $maxRestartCount) {
+        "$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss') - FFmpeg exited $restartCount times in $restartWindowMinutes minutes. Aborting recording." | Add-Content "$logDir\debug.log"
+        throw "FFmpeg exited repeatedly; aborting recording."
+    }
+
+    # Logarithmic backoff: first 10 restarts wait 1 second; beyond that delay grows slowly.
+    $restartDelay = if ($restartCount -le 10) { 1 } else { [math]::Round([math]::Log($restartCount - 9) * 5) }
+    "$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss') - FFmpeg exited unexpectedly. Restart $restartCount of $maxRestartCount. Waiting ${restartDelay}s before next attempt." | Add-Content "$logDir\debug.log"
+
+    Start-Sleep -Seconds $restartDelay
+}

Scripts/iKAT/Invoke-FFmpegCapture/log.md

@@ -21,10 +21,14 @@ Thursday, March 26, 2026 @ 1:02 PM
 Changes have been made, implementing it on the iKat server now
 1:30 PM:
 Several issues discovered and fixed during server deployment:
+
 - Username typo corrected: dpurner → dpuerner in all scripts
 - .vbs launcher files had hardcoded dev machine paths; updated to C:\PowerShellScripts\Scripts\iKAT\Invoke-FFmpegCapture\
 - Task Scheduler was not running ffmpeg in the user's interactive desktop session. Fixed by registering tasks with LogonType Interactive and RunLevel Limited. RunLevel Highest (elevation) also blocked gdigrab from accessing the desktop on RDS.
 - Recording output was being cut off: the script was reading logical (DPI-scaled) pixel dimensions via SystemInformation.VirtualScreen and passing them to gdigrab, which expects physical pixels. This caused the capture to be cropped, missing the taskbar and right side of screen. Fixed by removing manual dimension detection and letting gdigrab auto-detect the full desktop, with a -vf scale filter to enforce even dimensions required by libx264.
 - Switched output format from MP4 to MPEG-TS (.ts). MP4 requires a trailer written at the end to be playable; if ffmpeg is force-killed at logoff the file is corrupt. MPEG-TS writes self-contained 188-byte packets continuously with no trailer needed, so the file is fully playable up to the last written frame even after an abrupt kill.
 - Recording was still losing 5-10 seconds at the end due to libx264's encoder lookahead buffer. With a 25-frame GOP, libx264 holds up to 5 seconds of frames in memory before flushing. Fixed by adding -tune zerolatency (disables lookahead/B-frames, each frame encodes and flushes immediately) and reducing GOP to -g 5 (1-second keyframe intervals at 5fps). Also added -flush_packets 1 to force an OS-level flush after every TS packet. Maximum data loss is now under 1 second.
 - Dina's task (iKAT-Record-Dina) re-registered with RunLevel Limited to match Joey's working configuration.
+Thursday, March 27, 2026 @ 10:00 AM:
+UAC restart loop implemented and verified. When gdigrab loses the desktop handle during a UAC secure desktop transition, FFmpeg exits with code 0 (clean exit, not a crash). The original fix checked for non-zero exit codes to decide whether to restart — this meant UAC-triggered exits were mistakenly treated as graceful logoffs and the loop broke. Fixed by replacing the exit-code check with an explorer.exe presence check: if explorer is still running, the user session is still active and FFmpeg is restarted regardless of exit code. Confirmed working on server: UAC prompt triggered two .ts segment files as expected.
+Restart delay replaced with logarithmic backoff: first 10 restarts wait 1 second (fast recovery for UAC transitions); restarts 11+ use Round(Ln(restartCount - 9) * 5) seconds, growing from ~3s at restart 11 to ~20s at restart 60. Max restart count raised from 5 to 60, window from 10 to 60 minutes.

Tests/iCat/Invoke-iKATRecording.Tests.ps1 → Tests/iKAT/Invoke-iKATRecording.Tests.ps1

@@ -1,7 +1,7 @@
 Describe 'Invoke-iKATRecording.ps1' {
 
     BeforeAll {
-        $sut = Resolve-Path "$PSScriptRoot\..\..\Scripts\iCat\Invoke-FFmpegCapture\Invoke-iKATRecording.ps1"
+        $sut = Resolve-Path "$PSScriptRoot\..\..\Scripts\iKAT\Invoke-FFmpegCapture\Invoke-iKATRecording.ps1"
     }
 
     It 'Should exit if current user does not match TargetUser' {

Tests/iCat/Remove-ExpiredRecordings.Tests.ps1 → Tests/iKAT/Remove-ExpiredRecordings.Tests.ps1

@@ -1,6 +1,6 @@
 Describe 'Remove-ExpiredRecordings.ps1' {
     BeforeAll {
-        $sut = Resolve-Path "$PSScriptRoot\..\..\Scripts\iCat\Invoke-FFmpegCapture\Remove-ExpiredRecordings.ps1"
+        $sut = Resolve-Path "$PSScriptRoot\..\..\Scripts\iKAT\Invoke-FFmpegCapture\Remove-ExpiredRecordings.ps1"
         $testDir = Join-Path -Path $env:TEMP -ChildPath "FFmpegTest_$(New-Guid)"
         New-Item -ItemType Directory -Path $testDir | Out-Null
     }
@@ -21,13 +21,13 @@ Describe 'Remove-ExpiredRecordings.ps1' {
         Assert-MockCalled Write-Warning -Times 1 -ParameterFilter { $Message -match "Directory '$escapedDir' does not exist" }
     }
 
-    It 'Should delete .mkv files older than DaysToKeep and leave new files intact' {
+    It 'Should delete .ts files older than DaysToKeep and leave new files intact' {
         # Create a 6-day-old file
-        $oldFile = New-Item -Path $testDir -Name "old_video.mkv" -ItemType File
+        $oldFile = New-Item -Path $testDir -Name "old_video.ts" -ItemType File
         $oldFile.LastWriteTime = (Get-Date).AddDays(-6)
 
         # Create a new file (1 day old)
-        $newFile = New-Item -Path $testDir -Name "new_video.mkv" -ItemType File
+        $newFile = New-Item -Path $testDir -Name "new_video.ts" -ItemType File
         $newFile.LastWriteTime = (Get-Date).AddDays(-1)
 
         # Create a 6-day-old file with a different extension (should be ignored)
@@ -53,8 +53,8 @@ Describe 'Remove-ExpiredRecordings.ps1' {
         $logExists | Should -Be $true
 
         $logContent = Get-Content $logFile
-        $logContent | Should -Match 'Successfully deleted: old_video.mkv'
-        $logContent | Should -Not -Match 'new_video.mkv'
+        $logContent | Should -Match 'Successfully deleted: old_video.ts'
+        $logContent | Should -Not -Match 'new_video.ts'
         $logContent | Should -Not -Match 'old_notes.txt'
     }
 }