Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add sparse datasets #70

Merged
merged 53 commits into from
Dec 17, 2021
Merged

Add sparse datasets #70

merged 53 commits into from
Dec 17, 2021

Conversation

q-posev
Copy link
Member

@q-posev q-posev commented Dec 7, 2021 •
edited
Loading

This PR adds I/O for sparse datasets (e.g. n-electron integrals).
The sparse data is stored as a tuple of n indices (each represented as an int) and a floating point value.

q-posev and others added 30 commits November 24, 2021 17:44
[WIP] text back end
c1e1176
[WIP] working generator for sparse functions
d3ba8f3 
+ remove trailing whitespaces
[WIP] add C test for sparse I/O in text back end
db4b66d
add dependencies on generator Python scripts
99a2348
replace offset_data arg with pointer arithmetics
83c926f
add Fortran interface for sparse data
c30c3a5
add assert to check the read values
f9fd18c
better error handling in TEXT back end
7afce77
fixed string length for file name + comments
acdf982
add functions to read the number of stored sparse elements
d2c95cd
add Fortran test for reading sparse data size
7e83e66
unit testing for sparse data in C
1bb8ae5
update gitignore
5d3fadd
use size_max values provided by the read_size functions
c520175
[WIP] more general templated variables for sparse data
ed3bde9 
+ renamed templated variables with std_dtype_in|out suffix into _format_scanf|printf for clarity
adapt the generator to work for an arbitrary number of indices in spa…
e340c65 
…rse dset
alternative way to check for EOF with fgets while reading
e5bde10
add a test of reading a chunk that can reach EOF
1ad20c1
refactor to set common dtype substitutions in a function
bf5c651
adapt Fortran test to proper indices for sparse data
46a3815
@scemama
Modified density matrices
61ea409
add the unit test for HDF5 back end
ddcfff0
[WIP] working write_ for chunked extensible (sparse) datasets
d001844
Merge branch 'add-sparse-datasets' of github.com:TREX-CoE/trexio into…
fcf84eb 
… add-sparse-datasets
fix generator for single floats
c17297c
@q-posev
Merge branch 'master' into add-sparse-datasets
3fc3a78
add read_size function to HDF5 back end
e774cb6
working HDF5 I/O for sparse indices
e00f3b1
add tests/trexio_f.f90 module file to CLEANFILES
69e52d3
better detailed dict
e3a1e51
q-posev and others added 23 commits December 9, 2021 16:23
add tests for I/O of floating point attributes
2731b5d
Merge branch 'add-sparse-datasets' of github.com:TREX-CoE/trexio into…
02a1350 
… add-sparse-datasets
implement shift of sparse indices to be one-based in Fortran
7afb53b
only shift indices that have been read to the buffer (EOF case)
9f5ee46
adapt HDF5 back end to be generated for an arbitrary number of indices
1c16b75
fix bug in the generation of n-index functions for TEXT
32192f5
better error handling for HDF5
070ef06
fix memory leak in the Fortran test
b90abcc
improve tests
ba758e9
write sparse values and compress int indices depending on the max num…
a8b251d 
… value
read dim variable of sparse dset to compress storage of indices in HDF5
2e99a14
@scemama
Merge branch 'master' into add-sparse-datasets
1514f6e
@scemama
Added documentation
0837da9
return the number of read elements if read_sparse reached EOF
fafb155
Merge branch 'add-sparse-datasets' of github.com:TREX-CoE/trexio into…
968cfaa 
… add-sparse-datasets
[WIP] refactoring; write_dset_sparse done
1c49c81
@scemama
Better documentation in examples
9b4d11b
@scemama
Merge branch 'add-sparse-datasets' of github.com:TREX-CoE/trexio into…
5605be7 
… add-sparse-datasets
refactoring done; moved common blocks into _hdf5_open_read_dset_sparse
314beb5
(inout) buffer size instead if (out) [Fortran]
daa5bdb
Merge branch 'add-sparse-datasets' of github.com:TREX-CoE/trexio into…
1e457c4 
… add-sparse-datasets
optimize storage of indices depending on the size_max variable [TEXT]
2639b76
fix warnings on MacOS
31ccd27
@q-posev q-posev merged commit 8ca74ff into master Dec 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Handle index/value structs
2 participants
@q-posev @scemama