Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File size utilities #764

Merged
merged 3 commits into from
May 30, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
6 changes: 6 additions & 0 deletions include/highfive/H5File.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,12 @@ class File: public Object, public NodeTraits<File>, public AnnotateTraits<File>
return details::get_plist<FileAccessProps>(*this, H5Fget_access_plist);
}

/// \brief Get the disk size of this file in bytes
size_t getDiskSize() const;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a wrapper for the function H5Fget_filesize. Could we use getFileSize instead?


/// \brief Get the tracked unused space of this file in bytes
size_t getUnusedSpace() const;

protected:
File() = default;
using Object::Object;
Expand Down
18 changes: 18 additions & 0 deletions include/highfive/bits/H5File_misc.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -127,4 +127,22 @@
}
}

inline size_t File::getDiskSize() const {
hsize_t sizeValue = 0;
if (H5Fget_filesize(_hid, &sizeValue) < 0) {
HDF5ErrMapper::ToException<FileException>(
std::string("Unable to retrieve size of file " + getName()));

Check warning on line 134 in include/highfive/bits/H5File_misc.hpp

View check run for this annotation

Codecov / codecov/patch

include/highfive/bits/H5File_misc.hpp#L133-L134

Added lines #L133 - L134 were not covered by tests
}
return sizeValue;
}

inline size_t File::getUnusedSpace() const {
hssize_t unusedSize = H5Fget_freespace(_hid);
if (unusedSize < 0) {
HDF5ErrMapper::ToException<FileException>(
std::string("Unable to retrieve unused space of file " + getName()));

Check warning on line 143 in include/highfive/bits/H5File_misc.hpp

View check run for this annotation

Codecov / codecov/patch

include/highfive/bits/H5File_misc.hpp#L142-L143

Added lines #L142 - L143 were not covered by tests
}
return static_cast<size_t>(unusedSize);
}

} // namespace HighFive
50 changes: 50 additions & 0 deletions tests/unit/tests_high_five_base.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -426,6 +426,56 @@ TEST_CASE("Test groups and datasets") {
}
}

#if H5_VERSION_GE(1, 10, 1)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason it's needed it so set the file space strategy. The newly wrapped functions were introduced in HDF5 1.6.x and don't need guards.

TEST_CASE("FileSizeAndUnusedSpace") {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please split the test in two. One for the untracked file and one for the tracked variant.

const std::string file_name_untracked("filesize_untracked_unused.h5");
const std::string file_name_tracked("filesize_tracked_unused.h5");
const std::string ds_path("/dataset");
const std::vector<int> data{13, 24, 36};
const size_t untracked_file_size = 2048;
const size_t untracked_unused_space = 600;
const size_t tracked_file_size = 1158;
const size_t tracked_unused_space = 103;


{
File file(file_name_untracked, File::ReadWrite | File::Create | File::Truncate);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI this can be reduced down to:

File file(file_name_untracked, File::Truncate);

It's consistent with the rest of the file so there's no reason to change it.

auto dset = file.createDataSet(ds_path, data);
}

{
FileCreateProps fcp;
fcp.add(FileSpaceStrategy(H5F_FSPACE_STRATEGY_FSM_AGGR, true, 0));
File file(file_name_tracked, File::ReadWrite | File::Create | File::Truncate, fcp);
auto dset = file.createDataSet(ds_path, data);
}

{
File file(file_name_untracked, File::ReadWrite);
CHECK(file.getUnusedSpace() == 0);
file.unlink(ds_path);
CHECK(file.getUnusedSpace() == untracked_unused_space);
}

{
File file(file_name_untracked, File::ReadWrite);
CHECK(file.getDiskSize() == untracked_file_size);
CHECK(file.getUnusedSpace() == 0);
}

{
File file(file_name_tracked, File::ReadWrite);
file.unlink(ds_path);
}

{
File file(file_name_tracked, File::ReadWrite);
CHECK(file.getDiskSize() == tracked_file_size);
CHECK(file.getUnusedSpace() == tracked_unused_space);
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As you already said, these will fail. Since we're not testing if HDF5 is correct, it might be enough to check that:

  • the file size is >0.
  • the free space is == 0 (when applicable).
  • the free space is > 0 and < file_size (when applicable).

}
#endif

TEST_CASE("Test extensible datasets") {
const std::string file_name("create_extensible_dataset_example.h5");
const std::string dataset_name("dset");
Expand Down