WinGetSourceCreator / IndexCreationTool cannot index multi-file manifests (only singleton manifests supported)

[JohnnyElvis]

Type: Bug / Feature request
Area: WinGetSourceCreator, IndexCreationTool (src/WinGetSourceCreator)

Background (for readers new to this area)

winget can install from an offline / private pre-indexed source: a signed
source.msix that bundles a SQLite index (index.db) of package manifests. The
winget-cli repo ships a helper library, WinGetSourceCreator (used by the
IndexCreationTool utility under src/), to build that source.msix from a
folder of manifests. You hand it a small LocalSource JSON that points at an
AppxManifest.xml and one or more manifest folders, and it produces the indexed,
signed package.

winget package manifests come in two shapes:

• Singleton - the entire package (identity + installer + locale) in one
  .yaml file (ManifestType: singleton).
• Multi-file - the standard shape used by the public
  microsoft/winget-pkgs repo: the
  package is split across several files, one folder per version - a version
  file (<id>.yaml), an installer file (<id>.installer.yaml), and one or more
  locale files (<id>.locale.<lang>.yaml), each with its own ManifestType.
  These files are only valid together; none is complete on its own.

The problem: WinGetSourceCreator can only build a source from singleton
manifests. Point it at real winget-pkgs multi-file manifests and index creation
fails - so you cannot build an offline source directly from upstream manifests
without first flattening every package into a singleton.

Environment

• Component: src/WinGetSourceCreator (built via the IndexCreationTool project)
• winget-cli: master @ 4638f90e (near v1.30.80-preview)
• OS: Windows 11 (build 26100), .NET SDK build of the tool

---

Prerequisite: #6426 / #4181 must be fixed first

This fix depends on
#6426 (re-file of
#4181). Today
CopyManifestFile fails to create destination subdirectories, so the
IndexCreationTool / WinGetSourceCreator path breaks before it ever reaches
index creation - the tool cannot be used at all for anything laid out in
subfolders (which includes every real multi-file manifest).

Both fixes are required for the end-to-end offline-source scenario:

1. WinGetSourceCreator: CopyManifestFile does not create destination subdirectories (re-file of #4181, unaddressed ~2 years) #6426 / IndexCreationTool.exe - Tryng to find not existing data in WorkingDirectory #4181 - create destination subdirectories in CopyManifestFile
   (prerequisite; without it the tool is unusable for subfoldered input).
2. This issue - index multi-file manifest folders in CreateIndex.

---

Summary

WinGetSourceCreator.WinGetLocalSource can only build a pre-indexed source
(source.msix) from singleton manifests. Feeding it manifests in the
standard multi-file layout used by the microsoft/winget-pkgs repository
(separate *.installer.yaml, *.locale.<lang>.yaml, and *.yaml version
files, one folder per version) fails. This makes it impossible to build a
local/offline source directly from real winget-pkgs manifests without first
converting them to singletons.

Repro

1. Take any real multi-file manifest set from winget-pkgs. Example:
   Microsoft.DotNet.AspNetCore.10, versions 10.0.0-10.0.5, each version in
   its own folder with the three standard files:

   manifests/
     10.0.0/
       Microsoft.DotNet.AspNetCore.10.yaml              # ManifestType: version
       Microsoft.DotNet.AspNetCore.10.installer.yaml    # ManifestType: installer
       Microsoft.DotNet.AspNetCore.10.locale.en-US.yaml # ManifestType: defaultLocale
     10.0.1/ ...
   

2. Create a LocalSource JSON that points at the manifests folder:

   {
     "AppxManifest": "C:\\src\\AppxManifest.xml",
     "WorkingDirectory": "C:\\src\\work",
     "LocalManifests": [ "C:\\src\\manifests" ]
   }

3. Run the index/source creator against that JSON (via IndexCreationTool, or by
   calling WinGetLocalSource.CreateFromLocalSourceFile("localsource.json")).

Expected: a source.msix whose index.db contains every version of the
package (the same manifests that winget validate accepts).

Actual: index creation throws
Failed to add all manifests in directory to index, because each individual
multi-file part is rejected as an incomplete manifest (details below). No
source.msix is produced.

The defect: CreateIndex adds every *.yaml individually, which only works for singletons

CreateIndex enumerates all *.yaml recursively and calls AddManifest(file, rel)
per file. A multi-file manifest's version/installer/locale files are not
self-contained, so YamlParser::CreateFromPath rejects each individual file with
APPINSTALLER_CLI_ERROR_MANIFEST_FAILED (0x8A150004, IncompleteMultiFileManifest,
YamlParser.cpp:301). The retry queue then exhausts and throws
Failed to add all manifests in directory to index.

Note winget validate --manifest <folder> succeeds on the same input, so the
manifests themselves are valid - only the source-creator path is affected.

(This surfaces only after the #6426 directory-creation fix is in place, since
without it the copy step fails first with Could not find a part of the path ….)

Proposed fix

YamlParser::CreateFromPath already merges a multi-file manifest when handed its
directory. So, in CreateIndex, add a directory containing multi-file
manifest parts (any non-singleton/merged ManifestType) as a single entry
(pass the directory to AddManifest); add self-contained singleton/merged files
individually. This preserves existing singleton behavior (e.g. the E2E
TestData\Manifests flat folder) while enabling the multi-file layout.

A working patch implementing exactly this is ready and verified locally. If the
team is willing to review it, we're happy to open a PR (which would also fold in
the one-line #6426 fix so the scenario works end-to-end).

---

Proposed patch

All changes are in src/WinGetSourceCreator/WinGetLocalSource.cs.
(The diagnostic-only LoggingInit call used while investigating is intentionally
omitted from the patch below.)

Prerequisite: the CopyManifestFile subdirectory-creation fix tracked by
#6426 /
#4181 must also be in
place for the end-to-end scenario. It is not duplicated here.

Add multi-file manifest directories as a whole; keep singletons per-file

Before - every *.yaml is added individually (breaks on multi-file parts):

            Queue<string> filesQueue = new(Directory.EnumerateFiles(
                this.workingDirectory, "*.yaml", SearchOption.AllDirectories));

After - classify each directory and feed the right entry to AddManifest:

            // A multi-file manifest is split across several files in a single directory
            // (version + installer + defaultLocale [+ additional locales]). WinGetUtil merges
            // those files only when AddManifest is given the containing DIRECTORY, so such a
            // directory must be added as a whole. A singleton (or merged) manifest is a single,
            // self-contained file and is added individually.
            Queue<string> filesQueue = new(GetManifestEntries(this.workingDirectory));
            while (filesQueue.Count > 0)
            {
                int currentCount = filesQueue.Count;

                for (int i = 0; i < currentCount; i++)
                {
                    string entry = filesQueue.Dequeue();
                    try
                    {
                        var rel = Path.GetRelativePath(this.workingDirectory, entry);
                        indexHelper.AddManifest(entry, rel);
                    }
                    catch
                    {
                        // Retry later: e.g. a package dependency not yet added to the index.
                        filesQueue.Enqueue(entry);
                    }
                }

                if (filesQueue.Count == currentCount)
                {
                    throw new InvalidOperationException("Failed to add all manifests in directory to index.");
                }
            }

New helpers (added to the same class):

        // Enumerates the manifest entries under the working directory to feed to AddManifest.
        // Multi-file manifests (a directory whose yaml files include a non-singleton ManifestType
        // such as version/installer/defaultLocale/locale) are returned as the directory itself, so
        // WinGetUtil merges the parts. All other yaml files are returned individually as singleton
        // (or merged) manifests. This supports both a flat directory of singletons and the nested
        // per-version directory layout used by real winget-pkgs manifests.
        private static IEnumerable<string> GetManifestEntries(string root)
        {
            var directories = new List<string> { root };
            directories.AddRange(Directory.EnumerateDirectories(root, "*", SearchOption.AllDirectories));

            foreach (var directory in directories)
            {
                var yamlFiles = Directory.EnumerateFiles(directory, "*.yaml", SearchOption.TopDirectoryOnly).ToList();
                if (yamlFiles.Count == 0)
                {
                    continue;
                }

                if (yamlFiles.Any(IsMultiFileManifestPart))
                {
                    yield return directory;
                }
                else
                {
                    foreach (var yamlFile in yamlFiles)
                    {
                        yield return yamlFile;
                    }
                }
            }
        }

        // Returns true if the manifest file is part of a multi-file manifest, i.e. its ManifestType
        // is anything other than singleton or merged (version, installer, defaultLocale, locale, ...).
        private static bool IsMultiFileManifestPart(string file)
        {
            foreach (var line in File.ReadLines(file))
            {
                var trimmed = line.Trim();
                if (trimmed.StartsWith("ManifestType:", StringComparison.OrdinalIgnoreCase))
                {
                    var type = trimmed.Substring("ManifestType:".Length).Trim();
                    return !type.Equals("singleton", StringComparison.OrdinalIgnoreCase)
                        && !type.Equals("merged", StringComparison.OrdinalIgnoreCase);
                }
            }

            // No ManifestType found: treat as a self-contained singleton manifest.
            return false;
        }

Why this is safe for existing callers

• Singleton input (e.g. the E2E TestData\Manifests flat folder) still classifies
  every file as a singleton and adds it individually - unchanged behavior.
• Only directories that actually contain multi-file parts are added as a directory,
  matching how YamlParser::CreateFromPath already merges them.
• The dependency retry queue is preserved.

---

Verification (local)

Built source.msix from the 6-version AspNetCore 10 multi-file manifest set.
index.db contains all six versions:

Microsoft.DotNet.AspNetCore.10  Microsoft ASP.NET Core Runtime 10.0  10.0.0
Microsoft.DotNet.AspNetCore.10  Microsoft ASP.NET Core Runtime 10.0  10.0.1
Microsoft.DotNet.AspNetCore.10  Microsoft ASP.NET Core Runtime 10.0  10.0.2
Microsoft.DotNet.AspNetCore.10  Microsoft ASP.NET Core Runtime 10.0  10.0.3
Microsoft.DotNet.AspNetCore.10  Microsoft ASP.NET Core Runtime 10.0  10.0.4
Microsoft.DotNet.AspNetCore.10  Microsoft ASP.NET Core Runtime 10.0  10.0.5

Crucially, the index carries fields that exist only in the multi-file
*.locale.<lang>.yaml part - proving AddManifest merged the version +
installer + locale files rather than indexing bare version stubs:

index.db field	value	originating manifest part
names	Microsoft ASP.NET Core Runtime 10.0	locale
norm_publishers	microsoft	locale
tags	.NET, ASP.NET Core, runtime, dotnet, web, …	locale
monikers	aspnetcore-10	locale

A singleton-only index path (the current behavior) cannot produce these rows for
multi-file input - it fails outright at AddManifest. The populated
locale-derived data confirms the merge is correct end-to-end.

The resulting source.msix was further validated as a live pre-indexed
Microsoft.PreIndexed.Package source: the package deploys, registers its
com.microsoft.winget.source / IndexDB app extension, and winget opens the
index and returns all six versions - i.e. the built source is consumable by a
real winget client, not just structurally valid.

Offer to contribute a PR

We have a working, locally-verified fix (the patch shown above) and are happy to
open a pull request implementing it - if a maintainer is willing to review it.
The PR would also fold in the one-line CopyManifestFile directory-creation fix
tracked by #6426 / #4181 so the offline-source scenario works end-to-end, and the
temporary diagnostic logging used during investigation would be removed first.

Please let us know if you'd like the PR, or if you'd prefer to address it
internally.