superseded_gap_ranges walks the object headers in a pack's gaps (the byte ranges no index entry covers) and returns the ranges compact then drops:
https://github.com/borgbackup/borg/blob/master/src/borg/repository.py#L506-L521
hdr = RepoObj.ObjHeader(*RepoObj.obj_header.unpack(hdr_data))
obj_size = hdr_size + hdr.meta_size + hdr.data_size
if hdr.magic != OBJ_MAGIC or offset + obj_size > gend:
break
if hdr.chunk_id in chunks:
entry = chunks[hdr.chunk_id]
if entry.pack_id != pack_id or entry.obj_offset != offset:
drop_ranges.append((offset, obj_size))
offset += obj_sizemagic, chunk_id, meta_size and data_size all come from the header as it is, nothing authenticates them. (offset, obj_size) is then a range of bytes that gets deleted.
chunk_id being wrong is not much of a worry, it is 32 bytes and it has to hit an id that is in the index. meta_size/data_size are the problem, because they decide both how many bytes are dropped and where the next header is read:
- If
data_size is too large in a header whose chunk_id is in the index elsewhere, the dropped range extends past the object into the gap bytes behind it, and those are dropped without ever being looked at.
- If it is wrong in any header, the walk continues at a wrong offset and the rest of the gap is interpreted as objects at the wrong boundaries.
The gap bytes are unreferenced by the index, so normally dropping them is the point. It matters when the index does not describe everything that is in the pack, which is the situation gaps exist for in the first place: a backup that crashed before writing its index (#9868), or an index that is stale. Then a gap can hold the only copy of a chunk, and it can be dropped as collateral of a neighbouring object's corrupt length field.
#10083 removed exactly this kind of trust from PackReader.iter_headers (validate magic, version and that the object fits, raise otherwise). superseded_gap_ranges does the same walk on a destructive path and was not changed, which is inconsistent. At minimum it could reuse PackReader._parse_header so a bad header ends the walk over that gap instead of producing a drop range, and it could require the dropped object's size to agree with the index entry for that chunk id, which is available right there (entry.obj_size) and comes from a source that is not the bytes being examined.
superseded_gap_rangeswalks the object headers in a pack's gaps (the byte ranges no index entry covers) and returns the rangescompactthen drops:https://github.com/borgbackup/borg/blob/master/src/borg/repository.py#L506-L521
magic,chunk_id,meta_sizeanddata_sizeall come from the header as it is, nothing authenticates them.(offset, obj_size)is then a range of bytes that gets deleted.chunk_idbeing wrong is not much of a worry, it is 32 bytes and it has to hit an id that is in the index.meta_size/data_sizeare the problem, because they decide both how many bytes are dropped and where the next header is read:data_sizeis too large in a header whosechunk_idis in the index elsewhere, the dropped range extends past the object into the gap bytes behind it, and those are dropped without ever being looked at.The gap bytes are unreferenced by the index, so normally dropping them is the point. It matters when the index does not describe everything that is in the pack, which is the situation gaps exist for in the first place: a backup that crashed before writing its index (#9868), or an index that is stale. Then a gap can hold the only copy of a chunk, and it can be dropped as collateral of a neighbouring object's corrupt length field.
#10083 removed exactly this kind of trust from
PackReader.iter_headers(validate magic, version and that the object fits, raise otherwise).superseded_gap_rangesdoes the same walk on a destructive path and was not changed, which is inconsistent. At minimum it could reusePackReader._parse_headerso a bad header ends the walk over that gap instead of producing a drop range, and it could require the dropped object's size to agree with the index entry for that chunk id, which is available right there (entry.obj_size) and comes from a source that is not the bytes being examined.