Add FFmpeg agentic workflow with setup guide and example by Copilot · Pull Request #1931 · github/gh-aw

Copilot · 2025-10-18T06:39:18Z

This PR adds comprehensive FFmpeg support for GitHub Agentic Workflows, including installation guides, best practices, and an example workflow extracted from the GenAIScript ffmpeg implementation.

📁 Files Added

1. Shared Configuration: `ffmpeg.md`

A focused 162-line guide providing:

Automatic installation: FFmpeg setup step in frontmatter that runs before workflows
Version tracking: Step output captures FFmpeg version for reference in workflows
Cache folder: Automatic creation of /tmp/gh-aw/ffmpeg for caching intermediate results
Timeout documentation: Documented 5-minute timeout for bash/ffmpeg operations with guidance on setting adequate workflow timeouts
Stable hash computation: Guide for creating SHA-256 hashes to identify unique ffmpeg operations for efficient caching
Bash tools: Pre-configured ffmpeg and ffprobe commands
Common operations reference: Audio extraction (MP3, Opus for transcription), keyframe extraction, scene detection, video conversion
Transcription optimization: Added libopus codec for speech-to-text audio extraction
Streamlined command examples with essential flag explanations

2. Video Analyzer Workflow: `video-analyzer.md`

A streamlined video analysis workflow that performs focused analysis including:

Scene detection with configurable thresholds (0.3-0.5 range)
Audio extraction in MP3 format at 192k bitrate
Automated reporting via GitHub issue creation with detailed results

The workflow has been simplified to always perform full analysis (scenes + audio) without requiring users to choose between different analysis modes.

Usage:

gh workflow run video-analyzer.md \
  --field video_url="https://example.com/video.mp4"

🎯 Key Features from GenAIScript

Extracted and documented best practices from the GenAIScript implementation:

Scene Detection:

Threshold-based detection: select='gt(scene,threshold)'
Sensitivity range: 0.1 (very sensitive) to 0.6 (major changes only)
showinfo filter for timestamp logging

Audio Extraction:

MP3 with high quality: libmp3lame codec at 192k bitrate
Transcription optimization: libopus codec with mono channel, 12k bitrate, and voip application mode for optimal speech-to-text processing (documented in shared guide)

Automatic Installation:

FFmpeg installation step in frontmatter runs automatically before workflow execution
Version captured as step output: accessible via ${{ steps.setup-ffmpeg.outputs.version }}
Cache folder /tmp/gh-aw/ffmpeg created for storing intermediate results
No manual installation required in workflow prompts

Timeout Management:

Individual bash commands have a 5-minute timeout
Recommended workflow timeout of 15+ minutes for video processing operations
Guidance provided for handling longer operations

Caching Strategy:

Stable hash computation using SHA-256 of ffmpeg command + input files
Enables efficient caching by creating unique identifiers for each operation
Avoids reprocessing identical video operations

Bash Tools Configuration:

Pre-configured ffmpeg and ffprobe commands in frontmatter
Allows agents to execute ffmpeg operations seamlessly

🔧 Usage in Custom Workflows

Import the shared guide in any workflow:

---
on: workflow_dispatch
engine: copilot
imports:
  - shared/ffmpeg.md
timeout_minutes: 15  # Recommended for video processing
---

# Your Custom Video Workflow

FFmpeg and ffprobe are automatically installed via the import.
A cache folder at `/tmp/gh-aw/ffmpeg` is available for intermediate results.
Use stable hashes to cache expensive ffmpeg operations.
Access the FFmpeg version via ${{ steps.setup-ffmpeg.outputs.version }}.

Note: Bash commands have a 5-minute timeout. For longer operations,
break them into smaller steps or increase workflow timeout_minutes.

✅ Validation

All workflows compile successfully with strict mode
Unit tests pass (make agent-finish)
No linting errors
Workflows appear correctly in gh aw status
Imports work correctly and content is included in compiled lock files
FFmpeg installation step automatically included in compiled workflows
Version output properly captured in step outputs
Cache folder creation verified in compiled workflows
Bash tools configuration properly integrated
Stable hash guidance included in compiled prompts
Timeout documentation included in compiled workflow prompts

Original prompt

Generate an agentic workflow for ffmpeg:

provides steps to install in actions (there is probably a custom action)
prompt to help the agent find ffmpeg
Extract tips and tricks from the implementation of ffmpeg jn genaiscript: https://github.com/microsoft/genaiscript/blob/main/packages/core/src/ffmpeg.ts

best code to extract sound, usage of threshold, etc.

💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

github-actions · 2025-10-18T06:47:22Z

Agentic Changeset Generator triggered by this pull request.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2025-10-18T11:32:44Z

@@ -0,0 +1,170 @@
+# FFmpeg Agentic Workflows


@copilot delete file

Done. Deleted FFMPEG_README.txt file.

pelikhan · 2025-10-18T11:33:38Z

@@ -0,0 +1,269 @@
+---
+# FFmpeg Setup Instructions


@copilot rename to ffmpeg.md

Done. Renamed ffmpeg-setup.md to ffmpeg.md.

pelikhan · 2025-10-18T11:35:17Z

+sudo apt-get update && sudo apt-get install -y ffmpeg
+
+# Verify installation
+ffmpeg -version
+```


@copilot move as steps in the front matter

Done. Moved bash installation commands to frontmatter steps array. The Setup FFmpeg step now automatically runs before the agent.

pelikhan · 2025-10-18T11:35:58Z

+
+# FFmpeg Usage Guide
+
+## Installing FFmpeg


@copilot remove section as we are adding steps in the front matter to install ffmpeg

Done. Removed the "Installing FFmpeg" section since installation is now handled by the frontmatter steps.

pelikhan · 2025-10-18T11:38:18Z

+ffprobe -v error -select_streams v:0 -show_entries stream=width,height -of csv=s=x:p=0 input.mp4
+```
+
+## Best Practices


@copilot remained of file

Done. Removed the Best Practices and Error Handling sections to keep the file focused on essential operations. File reduced from 200 to 149 lines.

pelikhan

@copilot prompt to use /tmp/gh-aw/ffmpeg/ as cache folder. Updates steps to pre create this folder

…atter, add opus for transcription Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2025-10-18T11:56:26Z

+
+FFmpeg has been installed and is available in your PATH.
+
+## Finding FFmpeg


@copilot remove section

Done. Removed the "Finding FFmpeg" section.

pelikhan · 2025-10-18T11:56:53Z

+
+# FFmpeg Usage Guide
+
+FFmpeg has been installed and is available in your PATH.


@copilot add that /tmp/gh-aw/ffmpeg temp folder is available

Done. Added documentation that /tmp/gh-aw/ffmpeg temp folder is available for caching intermediate results.

pelikhan · 2025-10-18T11:57:49Z

+# Extract audio as WAV (uncompressed)
+ffmpeg -i input.mp4 -vn -acodec pcm_s16le output.wav
+
+# Extract audio as AAC
+ffmpeg -i input.mp4 -vn -acodec aac -ab 128k output.aac


@copilot remove section

Done. Removed WAV and AAC audio extraction examples, keeping only MP3 and Opus (for transcription).

pelikhan · 2025-10-18T11:59:21Z

+## References
+
+- FFmpeg Official Documentation: https://ffmpeg.org/documentation.html
+- FFmpeg Wiki: https://trac.ffmpeg.org/wiki
+- Opus Codec Documentation: https://opus-codec.org/


@copilot remove

Done. Removed the References section.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2025-10-18T12:08:00Z

@copilot prompt to use /tmp/gh-aw/ffmpeg/ as cache folder. Updates steps to pre create this folder

Done. Added /tmp/gh-aw/ffmpeg/ cache folder creation in setup steps and documented it in the usage guide prompt.

pelikhan · 2025-10-18T12:10:28Z

+
+# FFmpeg Usage Guide
+
+FFmpeg has been installed and is available in your PATH. A temporary folder `/tmp/gh-aw/ffmpeg` is available for caching intermediate results.


@copilot "ffmpeg and ffprobe" . Add version step output to specify the version.

Done. Updated to say "FFmpeg and ffprobe" and added step output to capture the version.

pelikhan · 2025-10-18T12:10:42Z

+  - name: Setup FFmpeg
+    run: |
+      sudo apt-get update && sudo apt-get install -y ffmpeg
+      ffmpeg -version


@copilot store version in a step output

Done. Added step id and version output using $GITHUB_OUTPUT. The version can be accessed via ${{ steps.setup-ffmpeg.outputs.version }}.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2025-10-18T12:17:25Z

+        description: 'URL to video file (must be publicly accessible)'
+        required: true
+        type: string
+      audio_format:


@copilot remove field, extract mp3 only

Done. Removed the audio-extractor workflow entirely as requested in the follow-up comment.

pelikhan · 2025-10-18T12:20:38Z

@@ -0,0 +1,174 @@
+---


@copilot remove sample

Done. Removed audio-extractor.md and audio-extractor.lock.yml.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2025-10-18T12:37:30Z

+        description: 'URL to video file to analyze (must be publicly accessible)'
+        required: true
+        type: string
+      analysis_type:


@copilot only support full analysis

Done. Removed analysis_type input field. Workflow now always performs full analysis including keyframes, scenes, and audio.

pelikhan · 2025-10-18T12:38:09Z

+## Important Notes
+
+### Performance Considerations
+- Process operations sequentially to avoid memory issues
+- Clean up intermediate files to save disk space
+- Monitor GitHub Actions runner resources
+- Consider file size limits for uploads/artifacts
+
+### Error Handling
+- Verify video download succeeded before processing
+- Check ffmpeg commands return success (exit code 0)
+- Validate output files exist and are not empty
+- Report any errors clearly in the issue
+
+### Best Practices from GenAIScript Implementation
+1. **Caching**: Store results to avoid reprocessing
+2. **Concurrency**: Process one video at a time
+3. **Formats**: Use JPG for frames (good quality/size balance)
+4. **Thresholds**: Scene detection works best at 0.3-0.5
+5. **Quality**: Use CRF 23 for good balance of quality/size
+6. **Timestamps**: Be precise with timestamp-based extraction
+7. **Size control**: Always specify output dimensions when needed
+
+### Security Considerations
+- Only process videos from trusted sources
+- Validate input URLs before downloading
+- Set reasonable timeout limits
+- Monitor disk space usage
+- Clean up temporary files after processing
+
+Good luck with your video analysis!


@copilot remove

Done. Removed the "Important Notes" section including Performance Considerations, Error Handling, Best Practices, and Security Considerations.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2025-10-18T12:48:34Z

+#### Keyframe Analysis:
+1. Extract all keyframes from the video:
+   ```bash
+   ffmpeg -i video.mp4 -vf "select='eq(pict_type,I)'" -fps_mode vfr -frame_pts 1 keyframe_%06d.jpg
+   ```
+2. Count the number of keyframes extracted
+3. Report keyframe distribution (approximately every N seconds)
+4. List the first 10 keyframe filenames with their timestamps


@copilot remove

Done. Removed the Keyframe Analysis section including I-frame extraction instructions.

pelikhan · 2025-10-18T12:49:09Z

+   # High quality MP3
+   ffmpeg -i video.mp4 -vn -acodec libmp3lame -ab 192k audio.mp3
+
+   # Uncompressed WAV
+   ffmpeg -i video.mp4 -vn -acodec pcm_s16le audio.wav


@copilot Only mp3

Done. Simplified audio extraction to MP3 only, removed WAV format.

pelikhan

@copilot apply comments

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Initial plan

03abee6

Copilot AI assigned Copilot and pelikhan Oct 18, 2025

Copilot started work on behalf of pelikhan October 18, 2025 06:39 View session

pelikhan marked this pull request as ready for review October 18, 2025 06:47

github-actions Bot and others added 3 commits October 18, 2025 06:49

Add changeset for ffmpeg workflow

3b16790

Add ffmpeg agentic workflows with setup guide and examples

4c9652f

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Add documentation for ffmpeg workflows

10516ef

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Add installation instructions and usage tips for ffmpeg~~ Add FFmpeg agentic workflows with comprehensive setup guide and examples Oct 18, 2025

Copilot AI requested a review from pelikhan October 18, 2025 07:02

Copilot finished work on behalf of pelikhan October 18, 2025 07:02

pelikhan reviewed Oct 18, 2025

View reviewed changes

Copilot started work on behalf of pelikhan October 18, 2025 11:39 View session

pelikhan reviewed Oct 18, 2025

View reviewed changes

Copilot AI and others added 2 commits October 18, 2025 11:49

Refactor ffmpeg shared config: rename file, add setup steps to frontm…

5eb8044

…atter, add opus for transcription Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Remove verbose sections from ffmpeg.md (Best Practices, Error Handling)

723969a

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~Add FFmpeg agentic workflows with comprehensive setup guide and examples~~ Add FFmpeg agentic workflows with setup guide and examples Oct 18, 2025

Copilot AI requested a review from pelikhan October 18, 2025 11:52

Copilot finished work on behalf of pelikhan October 18, 2025 11:52

pelikhan reviewed Oct 18, 2025

View reviewed changes

Copilot started work on behalf of pelikhan October 18, 2025 12:01 View session

Add ffmpeg cache folder, bash tools, and simplify ffmpeg.md

277d14d

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI requested a review from pelikhan October 18, 2025 12:09

Copilot finished work on behalf of pelikhan October 18, 2025 12:09

pelikhan reviewed Oct 18, 2025

View reviewed changes

Copilot started work on behalf of pelikhan October 18, 2025 12:12 View session

Add ffmpeg version output and stable hash section for caching

c0b3cae

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI requested a review from pelikhan October 18, 2025 12:20

Copilot finished work on behalf of pelikhan October 18, 2025 12:20

pelikhan reviewed Oct 18, 2025

View reviewed changes

Copilot started work on behalf of pelikhan October 18, 2025 12:21 View session

pelikhan reviewed Oct 18, 2025

View reviewed changes

Comment thread .github/workflows/shared/ffmpeg.md

Remove audio-extractor sample workflow

5edb2d1

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~Add FFmpeg agentic workflows with setup guide and examples~~ Add FFmpeg agentic workflow with setup guide and example Oct 18, 2025

Copilot AI requested a review from pelikhan October 18, 2025 12:27

Copilot finished work on behalf of pelikhan October 18, 2025 12:27

Copilot started work on behalf of pelikhan October 18, 2025 12:28 View session

Document 5-minute timeout for bash/ffmpeg operations

6c6d754

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan October 18, 2025 12:38

pelikhan reviewed Oct 18, 2025

View reviewed changes

Copilot started work on behalf of pelikhan October 18, 2025 12:38 View session

Simplify video-analyzer to only support full analysis

000bbba

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI requested a review from pelikhan October 18, 2025 12:48

Copilot finished work on behalf of pelikhan October 18, 2025 12:48

pelikhan approved these changes Oct 18, 2025

View reviewed changes

pelikhan reviewed Oct 18, 2025

View reviewed changes

Copilot started work on behalf of pelikhan October 18, 2025 12:50 View session

Remove keyframe analysis and simplify audio extraction to MP3 only

0c4db33

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan merged commit 58cfa23 into main Oct 18, 2025
4 checks passed

pelikhan deleted the copilot/add-ffmpeg-installation-guide branch October 18, 2025 12:57

Copilot AI requested a review from pelikhan October 18, 2025 12:57

Copilot finished work on behalf of pelikhan October 18, 2025 12:57


		FFmpeg has been installed and is available in your PATH.

		## Finding FFmpeg


		# FFmpeg Usage Guide

		FFmpeg has been installed and is available in your PATH.

Conversation

Copilot AI commented Oct 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📁 Files Added

1. Shared Configuration: ffmpeg.md

2. Video Analyzer Workflow: video-analyzer.md

🎯 Key Features from GenAIScript

🔧 Usage in Custom Workflows

✅ Validation

Uh oh!

github-actions Bot commented Oct 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 18, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 18, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 18, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 18, 2025

Choose a reason for hiding this comment

Uh oh!

pelikhan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 18, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 18, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 18, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI commented Oct 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 18, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 18, 2025

Choose a reason for hiding this comment

Copilot AI commented Oct 18, 2025 •

edited

Loading

1. Shared Configuration: `ffmpeg.md`

2. Video Analyzer Workflow: `video-analyzer.md`