Proposal: Standardize an offline index format

[jayconrod]

I propose we standardize a format for files generated outside of Gazelle that Gazelle can use as an index for dependency resolution.

This would help Gazelle resolve dependencies provided by external repos, which is currently not possible because Gazelle doesn't invoke Bazel and doesn't scan Bazel external repo directories. This would also help in situations where indexing is prohibitively expensive, such as when resolving an imported Java class to one of many vendored .jar files.

Related issues

• Index build files in external repositories #12
• Have gazelle run in external repos share its index with the gazelle run in the main project #2102
• Is there more convenient way to resolve external dependency? #669
• Can the API give us access to transitive dependency information? #2123

Proposed format

Index files are comma-separated value (CSV) files. This format is simple to generate, and relatively efficient to access using binary search.

The first row declares a list of file format properties as comma-separated values. For now, only the file format version is supported, but others may be added in the future. Unknown properties are ignored.

gazelle_index_version=1

The second row is a list of column names. The column names may appear in any order. Gazelle ignores columns with unknown names for forward compatibility. The following columns are supported.

srclang,implang,import,label

These have the same meaning as the arguments to # gazelle:resolve directives.

• srclang: language of the library being imported.
• implang: language of the code containing the import.
• import: the string by which the library is imported.
• label: Bazel label for the library.

All columns except implang are required. If the implang column is omitted, Gazelle acts as if it's present with all values equal to those in srclang. These columns are almost always the same anyway. They might be different when a rule for the source language generates bindings for several other languages with different import strings.

All remaining rows are import records.

Record rows must be sorted in lexicographic, ascending order by srclang, implang, then import. This helps Gazelle resolve imports without loading the entire file into memory.

Values may be quoted with single or double quotes (' or "). These have the standard meaning from CSV.

Values may contain the escape sequences \\ and \n. These expand to a literal backslash and newline respectively.

Quoting, escaping, and column order do not affect sorting or lookup operations. Values are unquoted and unescaped internally before being compared against other values. Raw bytes are not compared.

The newline character is reserved and may only be used at the end of each row. Newlines in values must be escaped. This allows Gazelle to find the beginning or end of a row from an arbitrary point within the file.

Example:

gazelle_index_version=1
srclang,import,label
go,github.com/bazel-contrib/bazel-gazelle/v2/errors,@gazelle//v2/errors
go,github.com/bazel-contrib/bazel-gazelle/v2/label,@gazelle//v2/label
go,github.com/bazel-contrib/bazel-gazelle/v2/rule,@gazelle//v2/rule

Usage

To use an index, a user may add one or more # gazelle:resolve_index directives to a BUILD file:

# gazelle:resolve_index cpp.gzlindex
# gazelle:resolve_index java.gzlindex

Each gazelle:resolve_index directive adds an index file path to the list of offline index files that may be searched to resolve dependencies in the current directory. Each path is a slash-separated path relative to the directory containing the BUILD file. This list is inherited by subdirectories. To reset the list, use an empty gazelle:resolve_index directive.

Offline index records are made available to extensions through resolve.RuleIndex.Find in the index passed to Resolve. Find shall only return records from offline index files if there are no records in the in-memory index, so indexed libraries in the repo take precedence over the offline index. When Find returns records from offline index files, it returns records from all files, not just the first matching file.

Extensions can decide whether to use the index at all and what precedence to apply. A typical order might be:

1. Standard library imports (not resolved to Bazel labels at all)
2. Overrides from # gazelle:resolve directives, found with resolve.FindRuleWithOverride
3. Index using RuleIndex.Find
4. Other language-specific conventions