Snapshot model weights/solver state to HDF5 files

[erictzeng]

This pull request enables Caffe to snapshot model weights and solver states to HDF5 files and makes this format the default. This format provides a number of advantages:

• It obeys weight-sharing, only snapshotting one copy of each of the parameters in the network. The old snapshotting method would save redundant copies of weight-shared parameters.
• This should enable snapshotting of networks that are arbitrarily large, whereas protobuf imposes a hard limit.
  • Note that, while snapshots themselves can be arbitrarily large, parameters themselves receive new constraints: the maximum number of dimensions an HDF5 dataset can have is 32, and each dimension is capped at 2^64. If anyone is using 33+ dimensional parameters, we can discuss further...

To avoid confusion with the old snapshotting methods, snapshotting to HDF5 files adopts new file extensions, namely .caffemodel.h5 and .solverstate.h5. When restoring either weights or solver history from a file, the extension of the file is checked. If the extension is .h5, it is loaded as an HDF5 file. All other extensions are treated as a binary protobuf file and loaded as before.

The default snapshot format is switched to HDF5 in this PR. If you prefer the old method, you can add snapshot_format: BINARYPROTO to your solver prototxt to restore binary protobuf snapshotting.

A few miscellaneous details:

• This PR is rebased off of one of @jeffdonahue's commits, primarily for its TestSnapshot test for gradient-based solvers.
• The few HDF5 helper functions that previously resided in util/io.cpp have been moved out to their own file, util/hdf5.cpp, and additional helper functions have been added.
• There were some nasty interface changes for both the Net and the Solver, since we now have methods for both BinaryProto and HDF5. Everything in Caffe checks out, but downstream users who have implemented their own non-SGD solvers/solvers with nonstandard snapshotting may have a bad time.

Potential caveats

• Commit d896647 changes the behavior of the function hdf5_save_nd_dataset. Previously, said function always saved 4-D blobs. It has since been changed to save N-D blobs instead. This could potentially break people's workflows if they were relying on HDF5OutputLayers to output 4-D blobs.
• Testing could be a bit more thorough. These are next on my list, but I wanted to throw this PR out there in the meanwhile. Off the top of my head:
  • There aren't any tests that compare the loaded solver history.
  • There aren't any tests that verify that weight-shared networks are correctly snapshotted/restored.

Possible extensions

These extensions won't end up in this PR, but possible things to do after this wraps up:

• It's probably worth looking into how to enable HDF5 compression, as that could potentially drastically reduce the size of serialized models.
• I like the idea of being able to include the network structure in the .h5 file itself, something that the binary protobuf snapshotting partially does, though it only captures the network topology and discards the layer-specific parameters. One possible way to do this is to write the prototxt as a string to an additional dataset in the .caffemodel.h5 file, though currently there's no complete way to turn a Net into a NetParameter, so this would require extra engineering effort.

[shelhamer]

This satisfies part of #1211.

[Yeongtae]

can i access the networks which are blobs using hdf5? If it can, please show the example.

[jeffdonahue]

I've skimmed through this and it mostly looks good, thanks @erictzeng. My one piece of feedback right now is that kMaxBlobAxes should be changed from INT_MAX to 32 unless/until it becomes possible to serialize blobs with more dimensions than that. (One possibility is that all blobs could be stored as 1D HDF5 datasets, with shapes themselves also separately stored as 1D HDF5 datasets, but I don't think anything like that needs to be done here; supporting >2GB nets should probably be considered higher priority than supporting blobs with more than 32 axes.)

[shelhamer]

Should this check equality? You might want to know for instance that the source layer has a bias but the target does not. Sorry, the check in 799-808 covers the rest.

[shelhamer]

This will be a good switch, and the backward compatibility saves a lot of heartache, but we might consider bringing the documentation and examples along with us as there are references to the current extensions here and there.

This looks good to me code-wise (once Jeff's comment is addressed) but you could squash related changes and fixes when you're done.

Since the weight sharing tests don't cover save and restore (TestSharedWeightsResume does not use an actual Solver) you could add a snapshot test with a simple weight shared net for completeness.

Thanks @erictzeng!

[jeffdonahue]

Since the weight sharing tests don't cover save and restore (TestSharedWeightsResume does not use an actual Solver) you could add a snapshot test with a simple weight shared net for completeness.

The tests I added in #2866 do cover this (though they're less unit tests and more integration tests than what you propose, as they also rely on the solver snapshot/restore correctness).

[shelhamer]

@jeffdonahue oh sweet, TestSnapshot and company take care of it then. I think this should be merged once kMaxBlobAxes is switched and the history squashed.

[bhack]

Why not https://google.github.io/flatbuffers/?

[shelhamer]

@bhack this lets us keep the same dependencies and interface for defining models. Migrating away to protobuf for a new format needs a good argument and its own issue since model definitions would change.

[bhack]

@shelhamer Flatbuffers support .proto parsing for easier migration from Protocol Buffers

[erictzeng]

That should be all comments addressed! The constant has been lowered to 32 as requested, and history has been squashed. Let me know if anything else seems off.

@Yeongtae I'm not sure I fully understand what you're asking, but this PR allows you to access network parameters via HDF5, if that's what you want. The parameters are stored in a fairly simple structure. Here's how you'd peek at the conv1 parameters in lenet:

$ h5ls examples/mnist/lenet_iter_10000.caffemodel.h5/data/conv1
0                        Dataset {20, 1, 5, 5}
1                        Dataset {20}

The datasets 0 and 1 correspond to the weights and biases of the layer, respectively.

[jeffdonahue]

shouldn't the diff dataset only be created if write_diff is set?

[jeffdonahue]

Everything looks good, thanks Eric!

[bhack]

My vote still go to flatbuffers as a natural google successor to protobuf. But with this merge hdf5 it is the de facto standard for caffe models now and nobody replied to the evaluation process of protobuff substitute.

[shaibagon]

What about a python interface for saving a net to HDF5? This can be useful for "net surgery".
I tried to hack it myself, adding

void Net_SaveToHDF5(const Net<Dtype>& net, string filename, bool write_diff) {
  net.ToHDF5(filename.c_str(), write_diff);
}

To python/caffe/_caffe.cpp, and .def("save_to_hdf5", &Net_SaveToHDF5) to the Net class definition.

However, I got this error:

HDF5-DIAG: Error detected in HDF5 (1.8.11) thread 139737896589120:
#000: ../../../src/H5G.c line 310 in H5Gcreate2(): unable to create group
major: Symbol table
minor: Unable to initialize object
#1: ../../../src/H5Gint.c line 194 in H5G__create_named(): unable to create and link to group
major: Symbol table
minor: Unable to initialize object
#2: ../../../src/H5L.c line 1638 in H5L_link_object(): unable to create new link to object
major: Links
minor: Unable to initialize object
#3: ../../../src/H5L.c line 1882 in H5L_create_real(): can't insert link
major: Symbol table
minor: Unable to insert object
#4: ../../../src/H5Gtraverse.c line 861 in H5G_traverse(): internal path traversal failed
major: Symbol table
minor: Object not found
#5: ../../../src/H5Gtraverse.c line 755 in H5G_traverse_real(): component not found
major: Symbol table
minor: Object not found
F0111 13:48:01.398217 28230 net.cpp:948] Check failed: layer_data_hid >= 0 (-1 vs. 0) Error saving weights to /path/to/model.h5.
*** Check failure stack trace: ***
Aborted (core dumped)

[shaibagon]

@shelhamer There seems to be some "hiccups" with snapshoting to hdf5 format.
See, for example, this SO question. I had similar issues myself.
Are you aware of these "hiccups"? are you working to solve them?

[shelhamer]

@shaibagon I'm not aware of any issue, so could you post an issue with details to reproduce the problem with Caffe master? I don't know anything about the OpenCV DNN package mentioned at that SO link.

Please mention @erictzeng in the issue as the author of this PR.