Fix h5ls SIGSEGVs in Nbit and Fletcher32 filter decode paths

[mattjala]

This PR fixes three related issues which could cause crashes when
crafted files were provided to h5ls. As it happens, all three are reachable from
any reader via H5Z_pipeline() during a chunked H5Dread(), so the root issues
affect the library itself.

• Nbit NULL/short parameter array (h5ls: SIGSEGV in Nbit filter processing when dumping datasets #6489, "Null_Pointer")

  H5Z__filter_nbit() dereferenced cd_values[0] before validating the
  parameter array. A file storing the filter with zero client-data
  values produced cd_values=NULL / cd_nelmts=0 and a NULL dereference.

  Fix is to reject a NULL cd_values or a cd_nelmts < 5 (the header at indices
  0..4 is always read) before any access.

• Nbit compressed-input buffer over-read (h5ls: SIGSEGV in Nbit filter processing when dumping datasets #6489, "Heap_Corruption_1")

  The H5Z__nbit_decompress* family walked the compressed input via
  index *j with no bound against the input size, and nbytes was never
  even passed in. Malformed parameters (e.g. d_nelmts=67108876 over a
  31-byte chunk) drove *j past the end, reading out of bounds in
  H5Z__nbit_decompress_one_byte().

  Fix is thread the input size through the whole decompress call chain,
  convert the void helpers (_one_byte, _one_nooptype, _one_atomic)
  to return herr_t, and bounds-check *j against buffer_size before every
  buffer[*j] read, propagating a filter error up the stack.

• Fletcher32 checksum length underflow (h5ls: SIGSEGV in H5_checksum_fletcher32 due to underflowed checksum length #6490, "Heap_Corruption_2")

  The read path computed src_nbytes = nbytes - FLETCHER_LEN with no
  lower-bound check. A corrupt 0-byte chunk underflowed src_nbytes to
  SIZE_MAX-3, which was passed straight to H5_checksum_fletcher32(),
  causing an out-of-bounds read.

  Fix is to reject nbytes<FLETCHER_LEN up
  front, since a valid Fletcher32 buffer must contain at least the
  trailing 4-byte checksum.

All three now fail gracefully with an error instead of crashing.

Also adds regression tests for the core library to guard against
the patched issues. The files used in the tests were produced with the new generator test/gen_bad_filters.c,
which writes each dataset via the public API (using earliest library
version bounds so the pipeline message lands in a version-1,
un-checksummed object header) and then patches the specific bytes.

Fixes #6489
Fixes #6490
Fixes #6488
Fixes #6492

[github-actions]

Review Checklist

This PR touches the following areas. Each needs a sign-off
from its listed owners before merging.

• src — @glennsong09
• test — @glennsong09
• *.cmake — @lrknox
• release_docs — @lrknox, @glennsong09

[Copilot]

Pull request overview

This PR hardens the HDF5 filter decode/read paths against crafted-file crashes by adding parameter/length validation to the N-Bit and Fletcher32 filters, and introduces regression tests (plus a generator) to ensure these malformed filter metadata cases fail gracefully instead of SIGSEGV/OOB reads.

Changes:

• Add early validation in the N-Bit decode path and thread compressed-buffer size through the N-Bit decompression helpers to prevent out-of-bounds reads.
• Add a minimum-length guard in the Fletcher32 read path to prevent checksum-length underflow.
• Add regression tests and a generator for malformed-filter test files, and register the new reference .h5 files in the test CMake setup.

Reviewed changes

Copilot reviewed 5 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/H5Znbit.c	Adds N-Bit parameter validation and bounds-checking during decompression to prevent crashes/OOB reads.
src/H5Zfletcher32.c	Rejects too-short Fletcher32 buffers to avoid size_t underflow and OOB reads.
test/dsets.c	Adds a regression test that verifies reads fail cleanly for crafted bad-filter metadata files.
test/gen_bad_filters.c	Adds a generator to create/patch malformed filter metadata files used by the regression test.
test/CMakeTests.cmake	Registers the new bad-filter .h5 reference files and adds the generator target.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[lrknox]

@mattjala This should probably have a CHANGELOG.md entry.

[mattjala]

This same set of changes also fixes #6488, since the root cause in both causes in the fletcher32 checksum-length underflow