Planning raw discs

Planning raw discs

Raw create builds one disc per run. Before that, it plans the complete source across discs, shows three alternative plans and lets you choose which units go onto this disc. The remaining units stay in the source for later runs. See Raw data discs for the overall workflow, recovery options and how the disc folder is built.

bd-archive create -s /data/incoming -n Streams-2026-10 -o /data/streams-2026-10 \
  --group-by depth:2 --order-by content-date --max-free 5 -b 25025314816

The capacity is an example; use the writable byte count for your media. Omit -b to detect capacity from an inserted disc, optionally using -D /dev/sr0. Planning stores no plan or history: each run plans what is in the source at that moment.

Choose indivisible units

--group-by decides what stays together on one disc. A unit is never split across discs, and relative paths are preserved below the disc's <name>/ folder.

--group-by	Unit
depth:1 (default)	Each direct subfolder of the source, kept whole with everything inside; each file directly in the source separately
depth:2 (or another depth)	Each folder two levels below the source, kept whole; each file above that level separately
depth:inf	Unlimited depth: each single file separately; folders are not kept together

For a layout such as channel/video/files, depth:1 keeps each channel together, depth:2 keeps each video with its companion files, and depth:inf lets companion files land on different discs. depth:inf behaves like depth:N with N beyond the deepest level of the source. Grouping folders above a unit, such as channel/ for depth:2, are recreated on the disc with their permissions and modification times.

Empty leaf directories are preserved as units when they are not already inside another unit. An entirely empty directory tree uses its newest directory mtime. Directory mtimes otherwise do not override file dates. Equal unit dates are ordered by relative path.

Units are never split silently. If one cannot fit including overhead, attachments and minimum recovery, choose finer grouping, larger media or a smaller -r, or use -m dar for a sliced archive across multiple discs. depth:inf can separate videos and their external subtitles or other companion files. Planning does not infer companion relationships from extensions, contact services, or change application metadata.

Choose the order

--order-by decides the order in which units fill the discs. Disc 1 of a plan, the disc this run creates, receives the first units in this order. A plan may depart from it to save a disc; the preview's Order consistency line reports how closely it follows the order.

--order-by	Order
name (default)	By relative path, sorted case-sensitively
mtime	By file modification time, oldest first
content-date	By recording/release date from file metadata, read with ExifTool, oldest first; files without one use their modification time

With mtime and content-date, a folder takes the size-weighted median date of its files: sorted by date, the file at which half of the folder's bytes is reached decides. A file holding more than half of the folder's bytes, such as a video beside its small companion files, decides alone. At an exact half-weight tie the earlier date wins; all-zero-byte groups use equal file weights. Several large files in one folder still yield a single date.

Order consistency is a percentage: 100% fills the discs strictly in order; lower values mean more units out of order or larger departures, measured in positions for name and in time for the date orders. With content-date, a Files with metadata line counts the files on this disc whose date came from content metadata, so unrecognized metadata shows up as a low count.

Content dates (--order-by content-date)

With --order-by content-date, dates are read with ExifTool, independently of the download application. For each file, planning prefers a recognized recording/original content date, then a release date, then the file modification time. The preview counts metadata dates and modification-time fallbacks. Unsupported files, missing or invalid dates, and files ExifTool cannot parse use modification time; a failed ExifTool invocation stops creation before any file is created.

Recognized fields include:

• Photos: EXIF DateTimeOriginal (including its offset/subseconds when available), XMP EXIF original dates, XMP Photoshop DateCreated, and IPTC DateCreated.
• Matroska: DATE_RECORDED, then DATE_RELEASED, then the common DATE tag. For example, DATE=20170504 resolves to 2017-05-04.
• QuickTime/MP4: explicit original dates and Apple's creationdate key, followed by release/content-date tags. Generic movie/track/media creation times are ignored.
• Audio: ID3 recording and original-release/release dates, and Vorbis DATE.

This is a conservative whitelist, not recognition of every date in every format. Container creation, muxing, encoding and tagging dates are excluded. Tag identity matters: ExifTool also calls Matroska's numeric DateUTC element DateTimeOriginal, but that element dates muxing and is deliberately ignored. Conflicting tags of equal priority use the earliest valid date. No dates are inferred from filenames, external JSON sidecars or embedded JSON attachments.

Complete dates such as 20170504, 2017-05-04 and EXIF date/time strings are accepted. Incomplete years/months and impossible dates fall back to the next candidate or modification time. Explicit offsets are normalized to UTC; values without an offset are interpreted as UTC. Preview dates are displayed in UTC. Existing filesystem timestamps and embedded metadata are never rewritten.

After a plan is selected, the preview shows the date range of the disc. Ranges cover recognized image/video/audio files within each unit, using their resolved dates (including modification-time fallbacks); units without recognized media use all their files. The planner compares unit medians, not every individual media timestamp. Thus 100% order consistency does not guarantee non-overlapping media ranges: an indivisible folder containing both 2017 and 2024 videos still spans those years. Choose finer grouping if that is undesirable.

Metadata is read in bounded batches without decoding video/audio or extracting embedded streams. Scanning is read-only and no metadata cache or plan is saved.

Three plans: efficient, balanced, ordered

The preview always shows the same three plans, one block per strategy:

Plan	Strategy	Method
1	efficient	Fills discs by size, largest units first; order counts only where it costs no disc
2	balanced	Fills discs in order and backfills remaining gaps with later units that fit
3	ordered	Never reorders: a new disc starts as soon as the next unit does not fit

Each plan covers the whole source; this run creates its first disc, the one holding the earliest unit in fill order. Each block lists:

Line	Meaning
Method	How the strategy fills the discs, as in the table above
Units	Units on this disc
Data	Data of the units on this disc
Name range / Date range	First and last unit by name, or the media date range (UTC)
Files with metadata	With content-date only: files on this disc dated from metadata
Free (unused)	Data budget left on this disc before --fill-recovery
Recovery (fixed)	Recovery data from -r, with its share of the disc's data
Recovery (fill)	With --fill-recovery only: recovery data added from the free space, or skipped below 0.5% of the disc's data
Recovery (total)	With --fill-recovery only: fixed and filled recovery data together
Estimated disc	Write size of the finished disc, including recovery data and the 1-MiB end margin
Remaining	Data that stays in the source after this disc
Further discs	Further discs of the plan, plus units deferred by --max-free
Order consistency	How closely the whole plan follows the fill order

The recovery lines are measured with the same sparse stand-ins and the same rule that creation uses, so they show the recovery data creation writes.

Because every run plans the source again, later runs usually continue the same plan, but new or removed files can change it. A plan may fall back to a more ordered packing when that needs no extra disc, so efficient never uses more discs than balanced, and balanced never more than ordered; plans can therefore be identical. Enter selects the first available plan. The prompt is skipped when the first discs of the available plans are identical or only one strategy produced a plan; a Selected plan N header names a chosen plan, and an Automatically selected plan header states why no choice was needed. With -y, the most ordered available plan is selected without a prompt.

The preview then shows the volume label and disc layout of the selected plan.

Defer the last units (--max-free)

Without --max-free, every unit is included in each plan.

With --max-free, the last units in fill order may stay in the source so that every disc of a plan is filled:

--max-free 5       at most 5 percent of each disc's data budget unused
--max-free 500M    at most 500 decimal MB unused on each disc
--max-free 2G      at most 2 decimal GB unused on each disc

Percentages may be 0–100, and decimal values such as 1.5G are accepted. The limit applies to every disc of a plan, including the disc this run creates. Minimum recovery, attachments, checksums, filesystem overhead and safety reservations are excluded from the data budget it applies to. Each strategy keeps the longest prefix whose plan meets the limit; a strategy without such a prefix shows no plan within the free-space limit. ordered is affected most, because strict order leaves gaps that no cutoff can fill. If no strategy qualifies, nothing is created. Deferral always leaves a suffix in fill order, never arbitrary earlier large items selected out in favor of later small ones.

Order consistency derives from a fixed blended score that weighs the affected-unit share (50%), average normalized distance (25%), and squared worst normalized distance (25%); only a plan without departures reaches 100%. There are no weight or shift knobs.

Planning limits

Planning is a deterministic bounded heuristic, not an exhaustive optimizer. efficient and balanced apply bounded boundary moves/swaps that reduce disorder without adding a disc. With --max-free, plans that meet the limit are preferred over plans with fewer discs that miss it, and moves/swaps never turn a plan that meets the limit into one that misses it. efficient additionally searches each cutoff for the most ordered plan whose every disc meets the limit, fewest discs first: exactly for up to 200 units within 20,000 search steps per cutoff, otherwise the best plan found within those steps. This finds tight packings, such as three nearly full discs, that size-first packing misses. All prefix lengths are searched for up to 64 units; larger backlogs use sampled cutoffs, recent cutoffs and sequential disc boundaries, and at most 64 candidate plans are measured per strategy. Conservative search weights and finite search can leave opportunities unused.

Offered discs are measured separately with mkisofs before anything is created; optimistic estimates are refined without splitting units. Sparse filesystem measurements reserve the payload, attachments, checksum text, the PAR2 index with the minimum recovery data, the README allowance and the 1-MiB end margin, with write size rounded to 32 KiB. They use metadata and temporary sparse stand-ins, without hashing the payload or generating PAR2. Sparse files require a temporary filesystem that supports their sizes; TMPDIR controls Python's temporary location. The one-set-per-disc PAR2 layout permits at most 32,768 non-empty files per disc; a unit exceeding that bound must be regrouped. With recovery enabled, units containing only empty files are planned together with non-empty data, unless attachments already provide non-empty data.

Next: Raw data discs · CLI reference.