WARC digest algorithm is hard coded to SHA-1 with no way to choose another

[rzo1]

What happens

WARCRecordFormat.getDigestSha1() computes a SHA-1 over the bytes and returns it as sha1:<base32>. That value is used for WARC-Payload-Digest and WARC-Block-Digest in response, request, resource and metadata records. There is no configuration key for the algorithm, so an operator who wants SHA-256 digests in their archives, which WARC 1.1 allows, cannot have them.

Where

external/warc/src/main/java/org/apache/stormcrawler/warc/WARCRecordFormat.java:105-112, used at :402, :404, :409, and reused by MetadataRecordFormat.java:106 and WARCRequestRecordFormat.java:85.

public static String getDigestSha1(byte[] bytes) {
    return "sha1:" + base32.encodeAsString(DigestUtils.sha1(bytes));
}

Why it matters

sha1: base32 is the convention across the WARC ecosystem, and the field identifies content for deduplication rather than authenticating it, so this is not urgent. It is still a fixed choice in a place where the bytes being digested come from the network, and operators whose own policy rules out SHA-1, or whose downstream tooling wants SHA-256, have no option other than rewriting the records. Making the algorithm a setting costs little and removes the argument.

Reproduction

Save as external/warc/src/test/java/org/apache/stormcrawler/warc/WARCDigestAlgorithmTest.java.

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to you under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.stormcrawler.warc;

import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;

import java.nio.charset.StandardCharsets;
import java.util.List;
import org.apache.storm.tuple.Tuple;
import org.apache.stormcrawler.Metadata;
import org.junit.jupiter.api.Test;

/**
 * Documents the current behaviour: the digest algorithm is hard coded to SHA-1 in
 * WARCRecordFormat.getDigestSha1 and there is no configuration key to pick another one. Once the
 * algorithm is configurable, this test should be replaced by one that sets the key and asserts the
 * prefix changes.
 */
class WARCDigestAlgorithmTest {

    @Test
    void digestPrefixIsAlwaysSha1() {
        String digest = WARCRecordFormat.getDigestSha1("some crawled bytes".getBytes(StandardCharsets.UTF_8));
        System.out.println("digest: " + digest);
        assertTrue(digest.startsWith("sha1:"), "digest algorithm is fixed to sha1");
    }

    @Test
    void metadataRecordBlockDigestIsAlwaysSha1() {
        Metadata metadata = new Metadata();
        metadata.addValue("source", "a source");
        Tuple tuple = mock(Tuple.class);
        when(tuple.getStringByField("url")).thenReturn("https://www.example.org/");
        when(tuple.getValueByField("metadata")).thenReturn(metadata);
        MetadataRecordFormat format = new MetadataRecordFormat(List.of("source"));
        String warcString = new String(format.format(tuple), StandardCharsets.UTF_8);
        System.out.println(warcString);
        assertTrue(
                warcString.contains("WARC-Block-Digest: sha1:"),
                "no configuration can change the algorithm used here");
    }
}

Run it:

mvn -pl external/warc test -Dtest=WARCDigestAlgorithmTest

It passes on main and pins the current behaviour, since there is no setting to assert against yet. Replace it after the fix with one that sets the key and checks the prefix changes.

digest: sha1:B7VTSBZ5WRVHOA4OQPESX36RFAJF2URB
...
Content-Type: application/warc-fields
WARC-Block-Digest: sha1:2O53BWA6NLGDTA4665VJAM6IJBA4C5DK

source: a source

[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

Suggested fix

Add a configuration key, for example warc.digest.algorithm, accepting sha1 and sha256, and thread it through WARCRecordFormat so MetadataRecordFormat and WARCRequestRecordFormat pick it up too. Replace the static getDigestSha1 helpers with an instance method that emits the matching sha1: or sha256: prefix, keeping the static ones for compatibility if anything outside the module calls them. Keep sha1 as the default: CDX indexes and revisit tooling downstream assume sha1: base32, and changing the default would break them. Decide the default separately from adding the option.

[abhinav-phi]

Hi! I'd like to take this up.
Plan: I'll make the WARC digest algorithm configurable (instead of hardcoding SHA-1), default to the current behavior for compatibility, document the new option, and add a test.
Could you please assign this to me? Thanks!

/assign
take

[rzo1]

Please also check #2034 if this is duplicated there.