std.io.Writer.Hashing + streamRemaining not correct if read/discard has happened

Zig Version

0.15.2

Steps to Reproduce and Observed Behavior

test "Old vs New vs Stream" {
    // dd if=/dev/urandom of=/tmp/test bs=1048576 count=4
    // old
    const sha = std.crypto.hash.sha3;
    var old: [32]u8 = undefined;
    {
        var e = sha.Sha3_256.init(.{});
        var read_buf: [4096]u8 = undefined;
        var file = try std.fs.openFileAbsolute("/tmp/test", .{});
        defer file.close();
        try file.seekBy(8); // Remove this in all cases and it works
        while (true) {
            const read = try file.readAll(&read_buf);
            if (read == 0) break;
            e.update(read_buf[0..read]);
        }
        e.final(&old);
    }

    // new
    var new: [32]u8 = undefined;
    {
        var e = sha.Sha3_256.init(.{});
        var read_buf: [4096]u8 = undefined;
        var file = try std.fs.openFileAbsolute("/tmp/test", .{});
        defer file.close();
        var reader = file.reader(&read_buf);
        try reader.interface.discardAll(8); // Remove this in all cases and it works
        while (true) {
            reader.interface.fillMore() catch |err| {
                switch (err) {
                    io.Reader.Error.EndOfStream => break,
                    else => return err,
                }
            };
            e.update(reader.interface.buffered());
            try reader.interface.discardAll(reader.interface.bufferedLen());
        }
        e.final(&new);
    }

    // stream ----> This is the problem case
    var stream: [32]u8 = undefined;
    {
        var read_buf: [4096]u8 = undefined;
        var file = try std.fs.openFileAbsolute("/tmp/test", .{});
        defer file.close();
        var reader = file.reader(&read_buf);
        try reader.interface.discardAll(8); // Remove this in all cases and it works
        var buffer: [config.BLOCK_SIZE]u8 = undefined;
        var e = std.io.Writer.Hashing(sha.Sha3_256).init(&buffer);
        _ = try reader.interface.streamRemaining(&e.writer);
        e.hasher.final(&stream);
    }

    try t.expect(mem.eql(u8, &new, &old));
    try t.expect(mem.eql(u8, &new, &stream));
}

error: 'scanner.test.Old vs New vs Hashing' 
Test.Old vs New vs Hashing (test)
    try t.expect(mem.eql(u8, &new, &stream));

Expected Behavior

If we are already a bit into the stream streamRemaining() (mentioned in #25173) will not work correctly. I added the implementation new which I am actually using, because it avoids double-buffering, too.

[Littleote]

The reason it's not working, is because you forgot to flush the remaining bytes from the writer.

The amended version would look like so:

    var stream: [32]u8 = undefined;
    {
        var file = try std.fs.openFileAbsolute("/tmp/test", .{});
        defer file.close();
        // No buffer necessary here as we are only discarding and streaming to a writer
        var reader = file.reader(&.{});
        try reader.interface.discardAll(8);
        var buffer: [4096]u8 = undefined;
        var e: std.Io.Writer.Hashing(sha.Sha3_256) = .init(&buffer);
        _ = try reader.interface.streamRemaining(&e.writer);
        // Flush last values from the buffer
        e.writer.flush() catch unreachable;
        e.hasher.final(&stream);
    }

With the test passing in 0.16.0-dev.205+4c0127566.

Thanks! I totally see my mistake and if there weren't many actual bugs because of writergate, I would have assumed more strongly that it is my mistake and eventually found the problem. I close this to avoid noise for the maintainers.

But I have a question that you @Littleote might be able to answer:

        // No buffer necessary here as we are only discarding and streaming to a writer
        var reader = file.reader(&.{});

I cannot do that, because in my actual code I do normal reading (which I like to be buffered) and hashing. It is a record-protocol. So I think the only option is to do what I do in // new, if I want to avoid double-buffering. Right?

[Littleote]

Reading the implementation, only the part of data that was already in the reader's buffer before streaming is copied (double-buffered), the rest is streamed directly into the writer's buffer.

Remember that you can join one of the zig communities to get help with other doubts.