Bug: `CorruptObjectException: DIRC checksum mismatch` when reading Git index written with `index.skipHash=true`

[yogurtearl]

Summary

Hattip to Copilot for helping to diagnose and propose a fix.

DirCache.readFrom() throws CorruptObjectException: DIRC checksum mismatch when reading a Git index file that was written with the index.skipHash=true configuration (available since Git 2.40). When this option is enabled, Git writes 20 zero bytes as the trailing checksum instead of computing a real SHA-1. The openrewrite/jgit fork always computes the real SHA-1 and compares it to the stored value, which fails because a real SHA-1 never equals all zeros.

Affected Version

• openrewrite/jgit: All releases up to and including v1.4.1 (commit 37244d51b, current main)
• Upstream eclipse/jgit: Fixed in commit 831da296d9c (2023-04-15, by Matthias Sohn) — added skipHash config support and NullMessageDigest

Environment for Reproduction

Component	Version
macOS	any (tested on macOS 26.3)
Git	2.40.0 or later (tested with 2.53.0)
Java	8+ (tested with Zulu 21.0.7)
Gradle	9.2.0 (ships with the repo wrapper)
openrewrite/jgit	v1.4.1 (37244d51b)

Root Cause

Git 2.40 (release notes) introduced the index.skipHash configuration option. When set to true (also enabled implicitly by feature.manyFiles=true — see C Git repo-settings.c), Git skips computing the SHA-1 checksum when writing the index file and instead writes 20 zero bytes as the trailing hash.

On the read side, C Git skips index checksum verification by default (verify_index_checksum is 0). When verification is forced (e.g., by git fsck), C Git additionally accepts all-zero hashes without error (read-cache.c).

Upstream eclipse/jgit added equivalent handling via an internal NullMessageDigest class and a three-way check:

// upstream eclipse/jgit (fixed in 831da296d9c)
if (!(skipHash
        || Arrays.equals(readIndexChecksum, hdr)
        || Arrays.equals(NullMessageDigest.getInstance().digest(), hdr))) {
    throw new CorruptObjectException(JGitText.get().DIRCChecksumMismatch);
}

The openrewrite/jgit fork does not have this check:

// openrewrite/jgit v1.4.1 (broken)
readIndexChecksum = md.digest();
if (!Arrays.equals(readIndexChecksum, hdr)) {
    throw new CorruptObjectException(JGitText.get().DIRCChecksumMismatch);
}

Since a real SHA-1 digest can never be all zeros, the comparison always fails when index.skipHash=true.

Workaround

If you cannot patch openrewrite/jgit, you can force Git to rewrite the index with a real checksum:

git config index.skipHash false
git update-index --force-write

This rewrites the index file with a real SHA-1 trailing hash, immediately unblocking jgit consumers. Note that subsequent git add/git commit operations will re-enable skipHash if feature.manyFiles=true is still set in your config.

Steps to Reproduce

Prerequisites

# Install Git 2.40+ (macOS with Homebrew)
brew install git
git --version   # must be >= 2.40.0

# Install Java 8+ (any distribution)
java -version   # e.g., openjdk 21.0.7

Reproduce via shell commands

# 1. Create a temporary test repo
TMPDIR=$(mktemp -d)
cd "$TMPDIR"
git init test-repo
cd test-repo

# 2. Enable index.skipHash (or feature.manyFiles which implies it)
git config index.skipHash true

# 3. Create a file and stage it
echo "hello world" > hello.txt
git add hello.txt

# 4. Verify the index has an all-zeros trailing checksum
xxd .git/index | tail -3
# Expected: last 20 bytes are all 00
# Example output:
# 00000060: ...  0000 0000 0000 0000 0000
# 00000070: 0000 0000 0000 0000 0000 00

# 5. Clone openrewrite/jgit and run the unit test
cd /tmp
git clone https://github.com/openrewrite/jgit.git
cd jgit

# 6. Copy the test file to the test directory
mkdir -p jgit/src/test/java/org/openrewrite/jgit/dircache
# Copy DirCacheSkipHashTest.java from the "Unit Test" section below

# 7. Run the test — it will FAIL with "DIRC checksum mismatch"
./gradlew :jgit:test --tests "org.openrewrite.jgit.dircache.DirCacheSkipHashTest"
# Note: requires Java 8+ on PATH; Gradle wrapper downloads automatically

Reproduce via Java API (minimal)

// Add openrewrite/jgit as a dependency:
//   group: org.openrewrite.tools, artifact: jgit, version: 1.4.1

import org.openrewrite.jgit.dircache.DirCache;
import org.openrewrite.jgit.util.FS;
import java.io.File;

// Point to any repo with index.skipHash=true
File indexFile = new File("/path/to/repo/.git/index");
DirCache dc = DirCache.read(indexFile, FS.DETECTED);
// ^^^ throws: CorruptObjectException: DIRC checksum mismatch

Stack Trace

Line numbers refer to the pre-fix version at commit 37244d51b:

org.openrewrite.jgit.errors.CorruptObjectException: DIRC checksum mismatch
    at org.openrewrite.jgit.dircache.DirCache.readFrom(DirCache.java:547)
    at org.openrewrite.jgit.dircache.DirCache.read(DirCache.java:406)
    at org.openrewrite.jgit.dircache.DirCache.read(DirCache.java:191)

Impact

Any tool using openrewrite/jgit to read a Git working directory's index will fail on systems where:

• index.skipHash=true is set explicitly, or
• feature.manyFiles=true is set (which implies index.skipHash=true in C Git), or
• A future Git version enables index.skipHash by default

This includes OpenRewrite recipe runners that scan local repositories, the Moderne CLI's mod build command, and any other consumer of the openrewrite/jgit library.

Proposed Fix

Just copy the full upstream fix from commit 831da296d9c (2023-04-15, by Matthias Sohn) ?

OR use this minimal, backwards-compatible version of the fix: skip checksum verification when the stored hash is all zeros.

Security note: When a null hash is accepted, no data integrity verification occurs for the index content. This is the same tradeoff C Git makes — when index.skipHash=true, the index is trusted without cryptographic verification. A genuinely corrupted index with non-zero garbage bytes will still be caught.

Diff

--- a/jgit/src/main/java/org/openrewrite/jgit/dircache/DirCache.java
+++ b/jgit/src/main/java/org/openrewrite/jgit/dircache/DirCache.java
@@ -543,7 +543,7 @@ private void readFrom(InputStream inStream) throws IOException,
 		}
 
 		readIndexChecksum = md.digest();
-		if (!Arrays.equals(readIndexChecksum, hdr)) {
+		if (!Arrays.equals(readIndexChecksum, hdr) && !isNullHash(hdr)) {
 			throw new CorruptObjectException(JGitText.get().DIRCChecksumMismatch);
 		}
 	}
@@ -578,6 +578,26 @@ private static boolean is_DIRC(byte[] hdr) {
 		return true;
 	}
 
+	/**
+	 * Check if the given hash is all zeros (a null hash). Git 2.40+
+	 * writes a null hash when {@code index.skipHash=true}, signaling
+	 * that no checksum verification should be performed.
+	 *
+	 * Note: this is a minimal read-path fix. The upstream eclipse/jgit
+	 * additionally reads the {@code index.skipHash} config and uses a
+	 * {@code NullMessageDigest} to skip SHA-1 computation entirely
+	 * (a performance optimization for very large indexes).
+	 */
+	private static boolean isNullHash(byte[] hash) {
+		if (hash.length != Constants.OBJECT_ID_LENGTH)
+			return false;
+		for (byte b : hash) {
+			if (b != 0)
+				return false;
+		}
+		return true;
+	}
+
 	/**
 	 * Try to establish an update lock on the cache file.

Scope and limitations

This is a minimal read-path fix. It does not include:

1. Write-side skipHash support: jgit will still always compute and write a real SHA-1 when writing the index. This means jgit-written indexes won't benefit from the skipHash performance optimization. This is a separate enhancement.

2. SHA-1 computation skip on read: The upstream eclipse/jgit uses NullMessageDigest to avoid computing SHA-1 over the entire index content when skipHash is configured. This fix still computes the full SHA-1 and discards the result. For very large indexes (millions of entries), porting NullMessageDigest would be a worthwhile follow-up optimization.

3. Config-based opt-in: The upstream fix reads index.skipHash from the repository config. This fix accepts null hashes unconditionally (matching C Git's default behavior where checksum verification is skipped entirely). Adding config-gated behavior would provide defense-in-depth but is not required for correctness.

Unit Test

File: jgit/src/test/java/org/openrewrite/jgit/dircache/DirCacheSkipHashTest.java

The test suite covers 6 scenarios:

Test	Description
readIndex_withNullChecksum_shouldSucceed	Single entry, null hash → should read successfully
readIndex_withCorruptChecksum_shouldThrow	Single entry, non-zero garbage hash → should throw
readIndex_withMultipleEntriesAndNullChecksum_shouldSucceed	3 entries in subdirs, null hash
readIndex_emptyWithNullChecksum_shouldSucceed	0 entries (32-byte file), null hash boundary case
readIndex_version4WithNullChecksum_shouldSucceed	V4 (path-compressed) empty index, null hash
readIndex_withTreeExtensionAndNullChecksum_shouldSucceed	Entries + TREE cache extension, null hash

Full test source (click to expand)

/*
 * Copyright (C) 2026, OpenRewrite and others
 *
 * This program and the accompanying materials are made available under the
 * terms of the Eclipse Distribution License v. 1.0 which is available at
 * https://www.eclipse.org/org/documents/edl-v10.php.
 *
 * SPDX-License-Identifier: BSD-3-Clause
 */
package org.openrewrite.jgit.dircache;

import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.openrewrite.jgit.api.Git;
import org.openrewrite.jgit.errors.CorruptObjectException;
import org.openrewrite.jgit.lib.Repository;
import org.openrewrite.jgit.util.FileUtils;

import java.io.File;
import java.io.RandomAccessFile;
import java.nio.file.Files;

import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;

/**
 * Tests that {@link DirCache} can read index files written with
 * {@code index.skipHash=true} (Git 2.40+), where the trailing 20-byte
 * SHA-1 is replaced with all zeros.
 *
 * <p>See: <a href="https://git-scm.com/docs/git-config#Documentation/git-config.txt-indexskipHash">
 * git-config index.skipHash</a></p>
 */
public class DirCacheSkipHashTest {

    private Repository db;
    private Git git;
    private File trash;

    @BeforeEach
    public void setUp() throws Exception {
        trash = Files.createTempDirectory("jgit-skiphash-test").toFile();
        git = Git.init().setDirectory(trash).call();
        db = git.getRepository();
    }

    @AfterEach
    public void tearDown() throws Exception {
        if (db != null) { db.close(); }
        if (trash != null) {
            FileUtils.delete(trash,
                    FileUtils.RECURSIVE | FileUtils.SKIP_MISSING);
        }
    }

    @Test
    public void readIndex_withNullChecksum_shouldSucceed() throws Exception {
        File hello = new File(trash, "hello.txt");
        Files.write(hello.toPath(), "hello world\n".getBytes());
        git.add().addFilepattern("hello.txt").call();

        File indexFile = new File(db.getDirectory(), "index");
        assertThat(indexFile).exists();

        DirCache before = DirCache.read(indexFile, db.getFS());
        assertThat(before.getEntryCount()).isEqualTo(1);
        assertThat(before.getEntry(0).getPathString())
                .isEqualTo("hello.txt");

        zeroTrailingChecksum(indexFile);

        byte[] raw = Files.readAllBytes(indexFile.toPath());
        byte[] trailer = new byte[20];
        System.arraycopy(raw, raw.length - 20, trailer, 0, 20);
        assertThat(trailer).isEqualTo(new byte[20]);

        DirCache after = DirCache.read(indexFile, db.getFS());
        assertThat(after.getEntryCount()).isEqualTo(1);
        assertThat(after.getEntry(0).getPathString())
                .isEqualTo("hello.txt");
    }

    @Test
    public void readIndex_withCorruptChecksum_shouldThrow() throws Exception {
        File hello = new File(trash, "hello.txt");
        Files.write(hello.toPath(), "hello world\n".getBytes());
        git.add().addFilepattern("hello.txt").call();

        File indexFile = new File(db.getDirectory(), "index");
        corruptTrailingChecksum(indexFile);

        assertThatThrownBy(() -> DirCache.read(indexFile, db.getFS()))
                .isInstanceOf(CorruptObjectException.class)
                .hasMessageContaining("DIRC checksum mismatch");
    }

    @Test
    public void readIndex_withMultipleEntriesAndNullChecksum_shouldSucceed()
            throws Exception {
        File dir = new File(trash, "src");
        dir.mkdirs();
        Files.write(new File(dir, "A.java").toPath(),
                "class A {}".getBytes());
        Files.write(new File(dir, "B.java").toPath(),
                "class B {}".getBytes());
        Files.write(new File(trash, "README.md").toPath(),
                "# Hello".getBytes());

        git.add().addFilepattern(".").call();

        File indexFile = new File(db.getDirectory(), "index");
        DirCache baseline = DirCache.read(indexFile, db.getFS());
        assertThat(baseline.getEntryCount()).isEqualTo(3);

        zeroTrailingChecksum(indexFile);

        DirCache after = DirCache.read(indexFile, db.getFS());
        assertThat(after.getEntryCount()).isEqualTo(3);
    }

    @Test
    public void readIndex_emptyWithNullChecksum_shouldSucceed()
            throws Exception {
        File indexFile = new File(db.getDirectory(), "index");
        byte[] emptyIndex = new byte[32];
        emptyIndex[0] = 'D'; emptyIndex[1] = 'I';
        emptyIndex[2] = 'R'; emptyIndex[3] = 'C';
        emptyIndex[7] = 2; // version 2
        Files.write(indexFile.toPath(), emptyIndex);

        DirCache dc = DirCache.read(indexFile, db.getFS());
        assertThat(dc.getEntryCount()).isEqualTo(0);
    }

    @Test
    public void readIndex_version4WithNullChecksum_shouldSucceed()
            throws Exception {
        File indexFile = new File(db.getDirectory(), "index");
        byte[] v4Empty = new byte[32];
        v4Empty[0] = 'D'; v4Empty[1] = 'I';
        v4Empty[2] = 'R'; v4Empty[3] = 'C';
        v4Empty[7] = 4; // version 4
        Files.write(indexFile.toPath(), v4Empty);

        DirCache dc = DirCache.read(indexFile, db.getFS());
        assertThat(dc.getEntryCount()).isEqualTo(0);
    }

    @Test
    public void readIndex_withTreeExtensionAndNullChecksum_shouldSucceed()
            throws Exception {
        File dir = new File(trash, "src");
        dir.mkdirs();
        Files.write(new File(dir, "Foo.java").toPath(),
                "class Foo {}".getBytes());
        Files.write(new File(trash, "build.gradle").toPath(),
                "apply plugin: 'java'".getBytes());

        git.add().addFilepattern(".").call();

        // Use jgit to write an index with TREE extension
        File indexFile = new File(db.getDirectory(), "index");
        DirCache cache = DirCache.lock(indexFile, db.getFS());
        try {
            cache.read();
            cache.getCacheTree(true); // forces TREE extension
            cache.write();
            cache.commit();
        } finally {
            cache.unlock();
        }

        DirCache baseline = DirCache.read(indexFile, db.getFS());
        assertThat(baseline.getEntryCount()).isEqualTo(2);

        zeroTrailingChecksum(indexFile);

        DirCache after = DirCache.read(indexFile, db.getFS());
        assertThat(after.getEntryCount()).isEqualTo(2);
        assertThat(after.getCacheTree(false)).isNotNull();
    }

    private static void zeroTrailingChecksum(File indexFile)
            throws Exception {
        try (RandomAccessFile raf =
                     new RandomAccessFile(indexFile, "rw")) {
            raf.seek(raf.length() - 20);
            raf.write(new byte[20]);
        }
    }

    private static void corruptTrailingChecksum(File indexFile)
            throws Exception {
        try (RandomAccessFile raf =
                     new RandomAccessFile(indexFile, "rw")) {
            raf.seek(raf.length() - 20);
            byte[] garbage = new byte[20];
            for (int i = 0; i < garbage.length; i++) {
                garbage[i] = (byte) (0xDE + i);
            }
            raf.write(garbage);
        }
    }
}

Test Results

Before fix (v1.4.1 as-is)

DirCacheSkipHashTest > readIndex_withNullChecksum_shouldSucceed() FAILED
    org.openrewrite.jgit.errors.CorruptObjectException: DIRC checksum mismatch

DirCacheSkipHashTest > readIndex_withCorruptChecksum_shouldThrow() PASSED

DirCacheSkipHashTest > readIndex_withMultipleEntriesAndNullChecksum_shouldSucceed() FAILED
DirCacheSkipHashTest > readIndex_emptyWithNullChecksum_shouldSucceed() FAILED
DirCacheSkipHashTest > readIndex_version4WithNullChecksum_shouldSucceed() FAILED
DirCacheSkipHashTest > readIndex_withTreeExtensionAndNullChecksum_shouldSucceed() FAILED

6 tests completed, 5 failed

After fix

DirCacheSkipHashTest > readIndex_withNullChecksum_shouldSucceed() PASSED
DirCacheSkipHashTest > readIndex_withCorruptChecksum_shouldThrow() PASSED
DirCacheSkipHashTest > readIndex_withMultipleEntriesAndNullChecksum_shouldSucceed() PASSED
DirCacheSkipHashTest > readIndex_emptyWithNullChecksum_shouldSucceed() PASSED
DirCacheSkipHashTest > readIndex_version4WithNullChecksum_shouldSucceed() PASSED
DirCacheSkipHashTest > readIndex_withTreeExtensionAndNullChecksum_shouldSucceed() PASSED

6 tests completed, 0 failed
BUILD SUCCESSFUL

References

• Git 2.40 release notes (introduced index.skipHash): https://github.blog/open-source/git/highlights-from-git-2-40/
• C Git read-cache.c (index reading logic): https://github.com/git/git/blob/v2.40.0/read-cache.c
• C Git repo-settings.c (feature.manyFiles → index_skip_hash): https://github.com/git/git/blob/v2.40.0/repo-settings.c
• Upstream eclipse/jgit fix commit: eclipse-jgit/jgit@831da29
• Git documentation for index.skipHash: https://git-scm.com/docs/git-config#Documentation/git-config.txt-indexskipHash