Internals

Internals

How the analysis and file operations actually work. This page is for understanding why RemuxForge behaves the way it does; nothing here is needed to use it.

Everything below is described from the implementation. Where a value is configurable it is named as it appears in Settings Reference.

Split: how a cut is made

Frame-perfect cutting of an inter-frame codec is the least obvious operation in RemuxForge, because a segment that starts mid-GOP cannot simply be copied: the frames between the requested start and the next keyframe are predicted from frames outside the segment.

Preparation, common to both paths

1. Chapters are read with mkvextract.
2. Presentation timestamps are extracted with mkvextract <file> timestamps_v2 0:<temp>, producing the per-frame PTS list. This comes from Source raw when set, otherwise from the input.
3. Frame count is counted with ffprobe and compared against the length of the PTS list. A mismatch aborts the job, because the two would no longer index the same frames.
4. Duration is read with ffprobe -show_entries format=duration.
5. Segments are built from the cut mode, then named from the template.
6. Video parameters are read with ffprobe: codec, pixel format, colour space, primaries, transfer, range. These are reused verbatim when a re-encode is needed.
7. If any segment's output path resolves to the input file, the job aborts before writing anything.

Fast path

Selected when Snap is not off and Source raw is either unset or equal to the input. Boundaries have already been moved to keyframes, so nothing needs re-encoding.

The frame-rate mode (CFR or VFR) is detected first; if it cannot be determined the job stops rather than guessing. The cut is then mkvmerge --split parts:<start>-<end>.

There is one exception. mkvmerge will not split FLAC audio together with the video, so when a FLAC track is present the segment is built in two pieces: mkvmerge splits the video alone (--no-audio --no-subtitles --no-chapters), ffmpeg copies audio and subtitles for the same time range (-map 0:a? -map 0:s? -c:a copy -c:s copy -vn -avoid_negative_ts make_zero), and the two are muxed together.

Slow path

Selected when Snap is off (frame-perfect), or when Source raw points at a different file. The raw video bitstream is extracted, and ffprobe produces a frame map of byte offset, size and keyframe flag for every frame.

Timecodes. A timecode format v2 file is written for the segment, rebased so its first frame is time 0, with millisecond values taken from the source PTS list. This is what preserves VFR timing across the cut.

If the segment starts on a keyframe, the video bitstream is a plain byte-range copy from that frame's offset to the end of the last frame's data. No decoding at all.

If it does not, the segment is built in three pieces:

Piece	How it is produced
head	frames from the requested start up to (not including) the next keyframe, re-encoded all-intra
parameter sets	the original SPS/PPS/VPS, extracted from the raw bitstream
rest	byte-range copy from that keyframe to the end of the segment

Three details make this work:

• The head is re-encoded from the original container, not from an isolated raw fragment, so the decoder receives correct extradata and reference frames. The search for the previous keyframe determines how many frames are decoded and discarded before the range of interest.
• The search for the next keyframe is bounded (KEYFRAME_LOOKAHEAD, 500 frames). If none is found within the segment, the job fails rather than producing something wrong.
• The original parameter sets are re-injected between head and rest. After the re-encoded head the decoder holds the new encoder's parameter sets, which do not describe the untouched original frames that follow.

The re-encoded head is then probed: if it does not contain exactly the expected number of frames, the job fails.

Assembly. head + parameter sets + rest are concatenated into one bitstream, remuxed with mkvmerge -o video.mkv --timestamps 0:timecodes.txt video.bs, and the resulting frame count is checked against the expectation a second time. Audio and subtitles are copied for the time range with ffmpeg into a separate container, chapters are written to a Simple-format file rebased to the segment start (generic Chapter N names are renumbered), and a final mkvmerge combines video, AV and chapters.

If audio/subtitle extraction fails, the segment is muxed video-only with a warning rather than failing.

The slow path therefore decodes and re-encodes only the frames between the requested start and the next keyframe, and verifies the frame count twice along the way. It also extracts the full raw bitstream and builds a frame map for the whole file first, which the fast path does not do.

Shared geometry bootstrap

Frame-sync, Deep analysis and Speed correction establish picture geometry before measuring time. This prevents a different scan, crop, sample aspect ratio or small translation from being mistaken for a temporal mismatch.

1. FFmpeg supplies coded size, SAR and DAR for both files.
2. An explicit L:R:T:B analysis crop wins when present. Otherwise 24 native-resolution frames distributed across each file are inspected, non-informative frames are discarded, and the stable black borders define the active rectangle.
3. Informative, non-duplicate frames are sampled every three seconds within the first three minutes. CPU/OpenCV or Vulkan SIFT and homography RANSAC find secure cross-file matches; at least five independent, geometrically concordant matches are required.
4. The accepted homographies are reduced to axis-aligned X/Y scale and translation. A robust consensus rejects unrelated pairs, then pixel and gradient correlation refines the four parameters.
5. The intersection of the two active pictures becomes the common comparison viewport. The same crop, scale and translation are retained for subtitle canvas rewriting.

SIFT therefore answers only the bootstrap question, “which pixels describe the same picture geometry?”. Temporal alignment is measured separately. A small frame-to-frame gate weave does not become a different global transform: consensus is taken across independent matches, while the later dHash comparison operates at a much coarser spatial resolution.

Deep analysis also calibrates its central dHash viewport from independent windows. The SIFT consensus replaces that viewport only when the affine projection measurably repairs dHash agreement on the already-confirmed pairs.

Frame-sync: how the offset is found

Frame-sync produces one constant offset. After the shared geometry bootstrap it extracts compact horizontal and vertical dHashes on the real frame PTS.

Phase 1: initial delay

When both files contain a usable audio track in the same language, normalized audio-envelope correlation supplies the initial candidate. Audio is a fast way to locate the search corridor here; it does not replace the later visual verification. If that candidate fails the video checkpoints — for example because the two muxes assign different container delays to the shared audio track — Frame-sync runs the full visual search and repeats the checkpoints from the visual candidate. The video therefore remains authoritative.

Without a shared track, three dense visual windows centred at roughly 20%, 50% and 80% of the source are compared. Similar dHashes vote for their PTS difference. At least two measurable windows are required, and every measured window must agree on the same offset; disagreement indicates an edit rather than a weaker vote.

Phase 2: checkpoint verification and final offset

The candidate is tested at VideoSync.NumCheckPoints positions distributed through the episode, 9 by default. Each checkpoint decodes a six-second source window and the corresponding wider language corridor at full frame rate. A coarse and then fine dHash scan measures both the best offset and the fraction of source frames it explains.

An unmeasurable checkpoint is omitted. A well-supported checkpoint that reports another offset rejects the result, because the files do not have one constant relationship. At least FrameSync.MinValidPoints accepted checkpoints are required, their offsets must agree within the calibrated tolerance, and their median is the final delay.

Confidence is the mean explained-frame fraction of the accepted checkpoints. Frame-sync deliberately uses the CPU hash backend for these short windows: device startup and transfers would cost more than the dHash work itself. The selected vision backend still applies to the shared SIFT geometry bootstrap.

Deep analysis: how the edit map is built

Deep analysis produces a list of cut and insert operations describing how the language release differs from the source. The video dHash profile is the primary temporal measurement; geometry has already been established by the shared bootstrap.

Phase 1: one linear signal extraction

Each file is decoded once in PTS order into two 64-bit dHashes, luminance, a small grayscale thumbnail and the real timestamp of every frame. Manual stretch is applied to the language PTS before comparison. CPU and Vulkan backends calculate the same hashes and offset grids.

When the files expose a usable shared audio track, its energy envelope is extracted as optional corroborating evidence. Missing audio does not invalidate the visual pipeline.

Phase 2: the global offset scale

At fixed intervals across the complete source timeline, each source dHash searches the corresponding Language corridor. A temporal anchor is retained only when its best match is below the detection threshold and no second, temporally distinct match is equally plausible. Repeated frames, scrolling credits and other ambiguous pictures therefore do not vote. Each retained PTS difference is quantized on the Language frame grid.

A single global dynamic-programming pass assigns every anchor to one of the observed offset states. Exact and one-frame-adjacent evidence is rewarded, while every state transition is penalized. Consecutive equal states become one regime; one-frame phase jitter is absorbed and unsupported excursions between compatible neighbours do not become edits. A terminal regime is preserved only when at least two anchors support a real final change. Transitions between the resulting regimes become cut or insert candidates.

Phase 3: boundaries, duration and judgement

The provisional scale is first anchored by the constant offset that maximizes agreement over the whole file. For every candidate, full-rate frame distances then refine the transition between the before and after offsets. The search bracket expands when the best change point touches an edge. A nearby black run selects its start; exclusive frames and the nearest supported visual extreme can move a non-black boundary to the first frame owned by the new regime.

Operation duration is not inferred from a detector width. The offsets on the stable regions to both sides are independently centred with coarse and fine dHash scans, and their difference is quantized to an integer number of Language frames. Shared audio is optional corroboration: it can select the point inside an ambiguous black interval and can settle a one-frame duration ambiguity. For a duplicated source cadence only, a reliable audio measurement can veto a visual step whose offset does not move. Closed opposite-operation pairs remain determined by video so audio cannot distort their equal frame counts.

Transitions of at most one frame are discarded and the remaining offset scale is stitched back together. The complete EditMap is then re-anchored and measured over the entire file.

The map is accepted only if its ordered operations keep at least the calibrated fraction of sampled frames aligned. Coverage is therefore a whole-file validity check, not a timeout or a performance gate.

Backend execution

Advanced.VisionBackend selects CPU or Vulkan visual compute. For Deep analysis this covers dHash generation/grid scans and the SIFT/RANSAC geometry bootstrap. For Frame-sync it covers the geometry bootstrap, while its short temporal dHash windows intentionally stay on CPU. Speed correction uses the selected backend for its SIFT-based visual validation.

FFmpeg decoding is a separate choice. Hardware decode is used only when the user enabled Hardware Acceleration with a valid method; otherwise analysis stays on software decode. A selected backend is never silently replaced during a job.

Why an audio format is required

When Language audio is imported, Deep analysis requires an audio format and a scope of Lang or All for the complete batch. The requirement is decided from the configuration before the batch starts, and every Language track is rendered, including constant-delay episodes with zero operations.

Speed correction carries the same requirement, for the same reason. Both are timeline corrections that reach the audio samples. See Remux Synchronization.

Speed correction: known ratio and visual offset

Speed correction never infers a playback ratio from CFR/VFR metadata, nominal frame rates, default_duration, audio or container duration. The ratio is selected explicitly by the user.

The UI offers the six directional conversions between 23.976, 25 and 29.97 fps. The CLI remains more general and accepts any positive decimal or fractional --stretch-factor.

Once the ratio is known, the shared geometry bootstrap runs first, then the CPU or Vulkan SIFT matcher compares time-sampled frames using their real PTS. The service validates that the requested scale has sufficient visual support and resolves the constant offset; it does not compare alternative ratios.

How the factor is applied

The resolved ratio is converted into an FFmpeg atempo chain and rendered onto the imported Language audio tracks. A single atempo stage accepts a multiplier between 0.5 and 2.0, so ratios outside that range are expressed as a chain of stages whose product is the required value. A ratio within 0.0001 of 1.0 is treated as identity and produces no filter. The video is not re-encoded.

This is why speed correction requires an audio format and a scope of Lang or All when Language audio is imported: the correction is a render, not a container-level flag.

Why the ratio is explicit

Nominal frame rates do not establish playback speed reliably: VFR, bobbed video and soft telecine can expose different rates without requiring temporal stretching. Requiring an explicit direction avoids turning metadata differences into destructive audio changes.

Audio processing: the order of operations

The order matters because peak normalization measures a signal that later steps would otherwise change.

Per-track planning

Before any encoding, a plan is built for every audio track. The plan decides whether a track needs a render at all, and the reason appears verbatim in the detail panel's AUDIO PROCESSING section: a track already in the target format reports already compatible with configured audio processing, one excluded by the scope reports outside generic audio scope, and one with nothing to do reports no render.

The mandatory cases override the scope: a Deep analysis edit map or an active source fill forces a render regardless of whether a generic conversion was requested.

The plan also records each audio track's initial timestamp relative to the video timeline of the same file. A direct, unprocessed track keeps that timestamp through mkvmerge. When FFmpeg must render the track, RemuxForge materializes a positive origin as leading silence or a negative origin as a trim, because FFmpeg otherwise rebases decoded audio to zero. This per-track origin is composed with stretch, the Deep analysis edit map and the effective Frame-sync/manual delay; it is not substituted for any of them. A head cut consumes the native leading gap first and then continues into decoded samples, so the delay is neither lost nor applied twice.

The post-filter

Every render starts from a common filter chain:

• aformat=sample_fmts=flt normalises the internal working format to float.
• For AC-3, sample rate and channel layout are pinned to values the encoder accepts, resolved from the track's channel count.
• When 24bit to 16bit is on, aresample=resampler=soxr:precision=28:dither_method=shibata:osf=s16 performs the reduction.

Where the downsample happens, and why

Without peak normalization, the dither is part of the single render pass that produces the output.

With peak normalization, the chain becomes two passes, and the downsample is moved into the first pass:

1. Render to a temporary PCM file, with the dither applied.
2. Measure the peak on that temporary file.
3. Encode from the temporary file to the final format, applying the gain.

The dither shapes the noise floor and alters the peak. Measuring before it would target a level that the finished file does not have, so the measurement is taken on a signal that already includes it.

Measuring the peak

The temporary file is analysed with ffmpeg -af astats=metadata=0:reset=0 -f null -, and Overall.Peak_level is read from the output. The applied gain is target_dB - measured_peak_dB, and both values appear in the log.

Peak level is a sample maximum across the whole track. There is no dynamics processing and no LUFS target.

Deep analysis edit maps

When an edit map is present, the same two-pass structure applies, with the edit map rendered into the temporary file first. The peak is therefore measured on the assembled timeline, after cuts and inserts, rather than on the original track.

Source fill

Source fill concatenates a segment of the source track onto the imported track, and the two arrive with independent levels.

With peak normalization off, the concatenation is a single operation.

With it on, both tracks are normalised before they are joined:

1. Render the source track and the language track to separate temporary files.
2. Measure each peak independently and compute a gain for each.
3. Apply the two gains, then concatenate.

This is what removes the level step across the splice. A single gain computed on the joined file would take its peak from whichever half is louder, and leave the difference between the two halves in place.

For the plain case where the concatenation happens first, the peak is still measured after it, so the target applies to the finished track rather than to one of its parts.

When nothing happens

A source fill that is configured but never triggers, because no gap exceeded the threshold, reports source-fill configured, no fill above threshold. The track is then handled as an ordinary conversion, or skipped entirely if no conversion was requested.

Subtitle canvas rewrite

Imported subtitles carry their own coordinate space. When the donor's geometry differs from the output's, the coordinates have to be rewritten rather than the video rescaled.

Where the geometry comes from

The transform is built from the geometry Frame-sync or Deep analysis already produced, including any manual analysis crop. No separate ffmpeg geometry probe is run. With neither analysis enabled there is no transform to build, so the option is ignored with a warning.

The transform is validated before use: if either active area has a non-positive dimension the rewrite is skipped.

Per-track processing

Each supported track is processed independently:

1. If an earlier step already produced a temporary file for that track, most importantly the Deep analysis subtitle timeline edit, the rewrite starts from that file rather than from the original. Timeline edits therefore always precede canvas rewriting.
2. Otherwise the single track is extracted from the language file. The original container is never modified.
3. The format-specific rewriter produces a standalone, muxable file.
4. The output is validated before it replaces the track. Only then does the merge use it.

A failure at any step leaves the untransformed track in place and logs a warning. This is why the log matters here: a successful rewrite and a warned fallback produce the same output file list.

What each format rewrites

Format	Rewritten
PGS / SUP	PCS canvas size, object coordinates, object crop fields, WDS windows, and the ODS bitmap objects when scaling is required
ASS / SSA	PlayResX, PlayResY, optional layout resolution, style margins, style scale fields, \pos, \org, \move, rectangular and vector clips, drawing paths, and selected size-related override tags
VobSub (IDX/SUB)	IDX size, org, scale, align, timestamp/filepos entries, SPU SET_DAREA, SET_DSPXA, and the 2-bit RLE bitmap data when scaling is required

PGS. When the active areas differ in size the object bitmaps are decoded, scaled and re-encoded with palette indices preserved; no RGBA antialiasing or palette quantization is used, because either would alter the colours the original palette defines. If a display-set would overflow the output canvas but its bounding box fits, that display-set alone is clamped. If a bounding box is genuinely larger than the canvas, the track is left unchanged.

ASS/SSA. The rewrite happens in script space rather than raw pixel space. Missing PlayResX/PlayResY falls back to the analysed display size and is reported. ScaledBorderAndShadow is honoured, defaulting to modern libass behaviour when absent. Non-uniform scaling combined with rotation or shear override tags is treated as unsafe and skipped: a coordinate rewrite cannot express the intended affine transform.

VobSub. The .idx is the primary file and the .sub sidecar is kept in step. Each SUB block is rewritten according to the IDX file positions, and filepos entries are updated afterwards. When scaling is required, DVD SPU top and bottom fields are decoded, scaled as 2-bit indexed bitmap data and re-encoded. Unsupported SPU commands or an invalid PES size update leave the track unchanged.

Metadata: how the pipeline is evaluated

Metadata analysis is a simulation over an in-memory model, which is what makes the preview exact.

The snapshot model

Every scanned file carries two models: the original, built from MediaInfo at scan time and never modified, and the current, which the rules operate on.

Each analysis run starts by cloning the original over the current. This is what makes repeated analysis deterministic: re-running F6 after editing a rule produces the same result as running it on a freshly scanned file, with no accumulated state from the previous run.

Rules then execute in order, each seeing the state the previous ones left. This is the cumulative behaviour described in Metadata Presets, and it is why:

• a token without a prefix reads the current value, as modified so far;
• [original.*] reads the untouched snapshot, regardless of what earlier rules did.

Disabled rules are skipped without evaluating their conditions.

Change records

Each operation records a change with its before value, after value, the rule that produced it, and whether it requires a remux. The change list is the preview: the colour-coded diff and the hover-to-see-the-rule tooltip are both reading it.

Removed tracks are collected separately, so later rules do not operate on a track that an earlier rule has already dropped.

Execution mode

After all rules have run, the execution mode is derived from the accumulated change list:

• If there are no changes: NoOp.
• If any change is marked as requiring a remux: MkvMerge. Track removal always is, and so are individual fields flagged in the field registry as not expressible through header edits.
• Otherwise: CopyPropEdit when the output policy writes to a separate folder, PropEdit when it overwrites in place.

A single remux-requiring change therefore promotes the whole file, which is why one Remove track rule changes the cost of a batch so sharply.

Why the analysis goes stale

The execution mode depends on the output policy, and the change list depends on the preset and the input set. Changing any of them invalidates the computed result, so every analysed record is marked Stale rather than left showing a preview that no longer corresponds to the settings.

Next

• Split Mode · Remux Synchronization · Metadata Presets
• Settings Reference: the parameters named above
• Development: project layout, for reading the source