Pure Go HDF5 Library - Production-Ready for Scientific Computing

[kolkov]

We're excited to share that the HDF5 Go library has reached production-ready status with feature-complete read/write support and comprehensive validation against the official HDF5 test suite.

What is HDF5 Go Library?

A modern, pure Go implementation of the HDF5 file format that requires no CGo dependencies. This library enables Go developers to work with HDF5 files - the industry standard for storing and managing large scientific datasets - without relying on the C library.

Key Features

✅ Pure Go Architecture

• No CGo required - Simplified deployment and cross-compilation
• No C dependencies - Works anywhere Go works
• Modern Go idioms - Clean, idiomatic API design
• Native performance - Optimized for Go runtime

✅ Complete Read/Write Support

• All HDF5 layouts - Compact, contiguous, chunked datasets
• All datatypes - Integers, floats, strings, compounds, arrays, enums, references
• Compression - GZIP, Shuffle filter, Fletcher32 checksums
• Advanced features - Variable-length types, unlimited dimensions, dataset resizing
• Full linking - Hard links, soft links (symbolic), external links (cross-file)
• Attributes - Dense and compact storage with modification support

✅ Production Quality

• 98.2% official test suite pass rate - Validated against 433 files from HDF5 1.14.6
• 86.1% test coverage - Comprehensive unit and integration tests
• Zero linter issues - 34+ linters, strict code quality standards
• Cross-platform - Linux, macOS, Windows fully supported
• Battle-tested - Real-world usage in scientific computing projects

✅ Performance Optimizations

• Hyperslab selection - Efficient data slicing (10-250x faster than full reads)
• Chunk-aware reading - Reads only required chunks, not entire datasets
• Memory efficiency - Buffer pooling and smart memory management
• B-tree rebalancing - Four modes for optimal performance (Default, Lazy, Incremental, Smart)

Format Compatibility

• Read: HDF5 superblocks v0, v2, v3
• Write: HDF5 superblocks v0, v2
• Object headers: v1 (legacy HDF5 < 1.8) and v2 (modern HDF5 >= 1.8)
• Full backward compatibility - Reads files from any HDF5 version

Use Cases

Scientific Computing

// Read scientific datasets
file, _ := hdf5.Open("experiment_data.h5")
dataset, _ := file.Dataset("/measurements/temperature")
data, _ := dataset.Read()

Data Analysis

// Efficient partial reading with hyperslab selection
selection := hdf5.NewHyperslabSelection().
    Select(0, 1000, 10).  // Start, count, stride
    Build()
data, _ := dataset.ReadSelection(selection)  // 10-250x faster!

Data Storage

// Write complex structured data
fw, _ := hdf5.CreateForWrite("results.h5", hdf5.CreateTruncate)
type Result struct {
    Time    float64
    Value   float32
    Status  int32
}
results := []Result{{1.0, 42.5, 1}, {2.0, 43.1, 1}}
fw.CreateDataset("/results", results, nil)

File Organization

// Navigate hierarchical structure
file.Walk(func(path string, obj hdf5.Object) {
    switch v := obj.(type) {
    case *hdf5.Group:
        fmt.Printf("📁 %s (%d children)\n", path, len(v.Children()))
    case *hdf5.Dataset:
        fmt.Printf("📊 %s [%v]\n", path, v.Dataspace().Dims())
    }
})

Why Pure Go?

Traditional CGo-based libraries require:

• ❌ C compiler toolchain
• ❌ HDF5 C library installation
• ❌ Complex cross-compilation
• ❌ Platform-specific binaries
• ❌ Version compatibility management

Pure Go advantages:

• ✅ Single static binary
• ✅ go get - that's it
• ✅ Cross-compile for any platform
• ✅ Simplified CI/CD pipelines
• ✅ No version conflicts

Installation

go get github.com/scigolib/hdf5

That's it. No system dependencies, no C compiler, no configuration.

Quick Example

package main

import (
    "fmt"
    "log"
    "github.com/scigolib/hdf5"
)

func main() {
    // Open existing file
    file, err := hdf5.Open("data.h5")
    if err != nil {
        log.Fatal(err)
    }
    defer file.Close()

    // Read dataset
    ds, err := file.Dataset("/temperature")
    if err != nil {
        log.Fatal(err)
    }

    data, err := ds.Read()
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("Data: %v\n", data)
}

Documentation

We provide comprehensive documentation:

• Quick Start Guide - Get started in 5 minutes
• Reading Data Guide - Comprehensive reading guide
• Datatypes Guide - HDF5 to Go type mapping
• Performance Tuning - Optimization strategies
• Architecture Overview - How it works internally
• API Reference - Complete GoDoc
• 7+ Working Examples - Real-world code samples

Comparison with Other Solutions

Feature	HDF5 Go (Pure)	gonum/hdf5	go-hdf5/hdf5
Pure Go	✅ Yes	❌ CGo wrapper	✅ Yes
Reading	✅ Full	✅ Full	❌ Limited
Writing	✅ Full	✅ Full	❌ No
HDF5 1.8+	✅ Yes	⚠️ Limited	❌ No
Advanced Datatypes	✅ All	✅ Yes	❌ No
Test Suite Validation	✅ 98.2% (433 files)	⚠️ Unknown	❌ No
Installation	✅ go get	❌ Requires C lib	✅ go get
Cross-compile	✅ Easy	❌ Complex	✅ Easy
Maintained	✅ Active	⚠️ Slow	❌ Inactive

Real-World Usage

The library is already being used in production for:

• MATLAB v7.3 file processing - Reading complex numbers from nested groups
• Scientific data analysis - Processing large astronomical datasets
• Data conversion pipelines - HDF5 to other formats
• Cross-platform tools - Deployed on Linux servers, macOS workstations, Windows desktops

Roadmap

We're committed to long-term support:

• Current: v0.12.x maintenance (bug fixes, optimizations, minor features)
• Q3 2026: v1.0.0 LTS with 3+ years support guarantee
• Future: Thread-safety with mutexes, SWMR mode, advanced filters (LZF, SZIP)

See our complete roadmap for details.

Contributing

We welcome contributions! Whether it's:

• 🐛 Bug reports
• 💡 Feature requests
• 📝 Documentation improvements
• 🔧 Pull requests
• ⭐ Starring the project

Check out our Contributing Guide to get started.

Community

• GitHub Repository: https://github.com/scigolib/hdf5
• Discussions: https://github.com/scigolib/hdf5/discussions
• Issue Tracker: https://github.com/scigolib/hdf5/issues
• HDF Group Forum: https://forum.hdfgroup.org/

Acknowledgments

Special thanks to:

• The HDF Group for the HDF5 format specification and official test suite
• Professor Ancha Baranova for invaluable support
• Community contributors - apollo3zehn-h5, MATLAB project users, and all early adopters
• gonum/hdf5 for inspiration

Get Started Today

go get github.com/scigolib/hdf5

Read the Quick Start Guide and start working with HDF5 files in pure Go!

---

Project Status: Stable - Production-ready with feature-complete read/write support
License: MIT
Go Version: 1.25+

Built with ❤️ by the HDF5 Go community