Skip to content

Latest commit

 

History

History
222 lines (164 loc) · 9.97 KB

faq.rst

File metadata and controls

222 lines (164 loc) · 9.97 KB
Raw

FAQ

What datatypes are supported?

Below is a complete list of types for which h5py supports reading, writing and creating datasets. Each type is mapped to a native NumPy type.

Fully supported types:

Type Precisions Notes

Bitfield Integer Float

1, 2, 4 or 8 byte, BE/LE 1, 2, 4 or 8 byte, BE/LE, signed/unsigned 2, 4, 8, 12, 16 byte, BE/LE

Read as unsigned integers

Complex Compound Strings (fixed-length) Strings (variable-length) Opaque (kind 'V')

8 or 16 byte, BE/LE Arbitrary names and offsets Any length Any length, ASCII or Unicode Any length

Stored as HDF5 struct

Boolean Array

NumPy 1-byte bool Any supported type

Stored as HDF5 enum

Enumeration References

Any NumPy integer type Region and object

Read/write as integers
Variable length array Any supported type See Special Types <vlen>

Other numpy dtypes, such as datetime64 and timedelta64, can optionally be stored in HDF5 opaque data using opaque_dtype. h5py will read this data back with the same dtype, but other software probably will not understand it.

Unsupported types:

Type Status
HDF5 "time" type
NumPy "U" strings No HDF5 equivalent
NumPy generic "O" Not planned

What compression/processing filters are supported?

Filter Function Availability
DEFLATE/GZIP Standard HDF5 compression All platforms
SHUFFLE Increase compression ratio All platforms
FLETCHER32 Error detection All platforms
Scale-offset Integer/float scaling and truncation All platforms

SZIP

Fast, patented compression for int/float
  • UNIX: if supplied with HDF5.
  • Windows: read-only

LZF

Very fast compression, all types

Ships with h5py, C source available

What file drivers are available?

A number of different HDF5 "drivers", which provide different modes of access to the filesystem, are accessible in h5py via the high-level interface. The currently supported drivers are:

Driver Purpose Notes

sec2 stdio core family mpio

Standard optimized driver Buffered I/O using stdio.h In-memory file (optionally backed to disk) Multi-file driver Parallel HDF5 file access

Default on UNIX/Windows

What's the difference between h5py and PyTables?

The two projects have different design goals. PyTables presents a database-like approach to data storage, providing features like indexing and fast "in-kernel" queries on dataset contents. It also has a custom system to represent data types.

In contrast, h5py is an attempt to map the HDF5 feature set to NumPy as closely as possible. For example, the high-level type system uses NumPy dtype objects exclusively, and method and attribute naming follows Python and NumPy conventions for dictionary and array access (i.e. ".dtype" and ".shape" attributes for datasets, group[name] indexing syntax for groups, etc).

Underneath the "high-level" interface to h5py (i.e. NumPy-array-like objects; what you'll typically be using) is a large Cython layer which calls into C. This "low-level" interface provides access to nearly all of the HDF5 C API. This layer is object-oriented with respect to HDF5 identifiers, supports reference counting, automatic translation between NumPy and HDF5 type objects, translation between the HDF5 error stack and Python exceptions, and more.

This greatly simplifies the design of the complicated high-level interface, by relying on the "Pythonicity" of the C API wrapping.

There's also a PyTables perspective on this question at the PyTables FAQ.

Does h5py support Parallel HDF5?

Starting with version 2.2, h5py supports Parallel HDF5 on UNIX platforms. mpi4py is required, as well as an MPIO-enabled build of HDF5. Check out parallel for details.

Variable-length (VLEN) data

Starting with version 2.3, all supported types can be stored in variable-length arrays (previously only variable-length byte and unicode strings were supported) See Special Types <vlen> for use details. Please note that since strings in HDF5 are encoded as ASCII or UTF-8, NUL bytes are not allowed in strings.

Enumerated types

HDF5 enumerated types are supported. As NumPy has no native enum type, they are treated on the Python side as integers with a small amount of metadata attached to the dtype.

NumPy object types

Storage of generic objects (NumPy dtype "O") is not implemented and not planned to be implemented, as the design goal for h5py is to expose the HDF5 feature set, not add to it. However, objects picked to the "plain-text" protocol (protocol 0) can be stored in HDF5 as strings.

Appending data to a dataset

The short response is that h5py is NumPy-like, not database-like. Unlike the HDF5 packet-table interface (and PyTables), there is no concept of appending rows. Rather, you can expand the shape of the dataset to fit your needs. For example, if I have a series of time traces 1024 points long, I can create an extendable dataset to store them:

>>> dset = myfile.create_dataset("MyDataset", (10, 1024), maxshape=(None, 1024)) >>> dset.shape (10,1024)

The keyword argument "maxshape" tells HDF5 that the first dimension of the dataset can be expanded to any size, while the second dimension is limited to a maximum size of 1024. We create the dataset with room for an initial ensemble of 10 time traces. If we later want to store 10 more time traces, the dataset can be expanded along the first axis:

>>> dset.resize(20, axis=0) # or dset.resize((20,1024)) >>> dset.shape (20, 1024)

Each axis can be resized up to the maximum values in "maxshape". Things to note:

  • Unlike NumPy arrays, when you resize a dataset the indices of existing data do not change; each axis grows or shrinks independently
  • The dataset rank (number of dimensions) is fixed when it is created

Unicode

As of h5py 2.0.0, Unicode is supported for file names as well as for objects in the file. When object names are read, they are returned as Unicode by default.

However, HDF5 has no predefined datatype to represent fixed-width UTF-16 or UTF-32 (NumPy format) strings. Therefore, the NumPy 'U' datatype is not supported.

Exceptions

h5py tries to map the error codes from hdf5 to the corresponding Exception class on the Python side. However the HDF5 group does not consider the error codes to be public API so we can not guarantee type stability of the exceptions raised.

Development

Building from Git

We moved to GitHub in December of 2012 (http://github.com/h5py/h5py).

We use the following conventions for branches and tags:

  • master: integration branch for the next minor (or major) version
  • 2.0, 2.1, 2.2, etc: bugfix branches for released versions
  • tags 2.0.0, 2.0.1, etc: Released bugfix versions

To build from a Git checkout:

Clone the project:

$ git clone https://github.com/h5py/h5py.git
$ cd h5py

(Optional) Choose which branch to build from (e.g. a stable branch):

$ git checkout 2.1

Build the project. If given, /path/to/hdf5 should point to a directory containing a compiled, shared-library build of HDF5 (containing things like "include" and "lib"):

$ python setup.py build [--hdf5=/path/to/hdf5]

(Optional) Run the unit tests:

$ python setup.py test

Report any failing tests to the mailing list (h5py at googlegroups), or by filing a bug report at GitHub.