compact may fail to write useful list folder of dindex file

[ts678]

• I have searched open and closed issues for duplicates.

---

Environment info

• Duplicati version: 2.0.3.5 to 2.0.5.0 were the ends of the tested range. Looks like a very old bug.
• Operating system: Windows 10 Version 1909
• Backend: Local folder

Description

Compact may fail to write the list folder into the newly created dindex file created by a dblock merge.
I think the files in this folder are performance-enhancers, controlled by --index-file-policy, so the dlist file blocklists reference can read the blocklist from the dindex file without downloading its dblock.

Downloading dblock files for DB rebuild in Recreate or direct restore seems to go against the design intent, slows things down, and might get questions or complaints from people who observe carefully:

Repair is downloading dblock files

Note that this is a far more subtle performance reduction than the all-dblocks-downloaded situation.

Database recreate desperately needs improvement #4041 has one person who might be heading for downloading all dblocks, and one who might be hitting bad dindex files. See technical remarks there.

Steps to reproduce

1. Prepare four test files, which are variations of a 1024 text character file, intended to force all files to get a blocklist, and all blocks to be different. Line endings shouldn't matter. My notepad had none.

1more.txt
1 1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234

2more.txt
12 1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234

3more.txt
123 1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234

4more.txt
1234 1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234

2. Backup 1more.txt and 2more.txt, using --blocksize=1 KByte and --keep-versions=1, also without encryption and with --no-auto-compact true for step-by-step view and ease of watching dindex.
3. Add 3more.txt and 4more.txt into backup via checkbox editing or move into folder, then backup.
4. Remove 1more.txt and 3more.txt from the backup, then backup. Uncheck is slightly tidier than a delete-from-folder because it won't even need to output a dblock to contain folder info update.
5. Compact. This merges the two half-empty dblock files into a new dblock, and its new dindex file.

• Actual result:
  New dindex file has no list folder.
• Expected result:
  New dindex file has list folder with two files containing the blocklists for 2more.txt and 4more.txt.

I'm not sure if you'll notice, but if you're looking very carefully in the DB and filenames, you'll see that the list folder filenames are not exactly the Base64 encoding of the hash, but a character-safe variant:

duplicati/Duplicati/Library/Main/Volumes/IndexVolumeWriter.cs

Lines 75 to 77 in 52929e9

	//Filenames are encoded with "modified Base64 for URL" https://en.wikipedia.org/wiki/Base64#URL_applications,
	using (var s = m_compression.CreateFile(INDEX_BLOCKLIST_FOLDER + Library.Utility.Utility.Base64PlainToBase64Url(hash), CompressionHint.Noncompressible, DateTime.UtcNow))
	s.Write(data, offset, size);

One way to try to spot this problem "in the wild" is to look for compact in logs, then look at the dindex from that time. Or look for dindex files with no list folder, then see if it was by compact. Without known sources though, it's hard to know whether no list folder is from all small files thus no blocklists needed.

Screenshots

Debug log

[warwickmm]

I wonder if we're simply missing a call to IndexVolumeWriter.WriteBlockList somewhere?

[duplicatibot]

This issue has been mentioned on Duplicati. There might be relevant details there:

https://forum.duplicati.com/t/database-recreate-will-take-4-months/11827/3