Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add advanced HPC functionality based on PIO library #720

Closed

Conversation

edhartnett
Copy link
Contributor

@edhartnett edhartnett commented Dec 17, 2017

This PR contains the changes needed to get PIO working within the netCDF library.

This brings some very powerful new features for HPC. These features will be very helpful to weather modelers, climate modelers, and anyone else using netCDF on many-core supercomputers.

Documentation is good on a per-function level. Doxygen documentation of PIO is not enabled yet pending merging of another PR I have with many Doxygen changes. Once that PR is merged I will turn on PIO documentation in the Doxyfile.in file.

New functions are added to netcdf_par.h to support the new features. The PIO features are only available on builds which include both netCDF-4 and parallel.

Fixes #555.

@DennisHeimbigner
Copy link
Collaborator

Is there any chance of getting a design document?

@edhartnett
Copy link
Contributor Author

Yes, more documentation is coming. Now that I've got everything working, I'll get it documented. ;-)

@edhartnett
Copy link
Contributor Author

OK, there are now two .md files in the libpio directory: using_pio.md for a user-level overview, and developer_docs.md for developer level documentation. These documents will develop over the coming days.

@edhartnett
Copy link
Contributor Author

@DennisHeimbigner take a look at the two docs I added, and most importantly the example examples/C/pio_simple_async.c. This example shows the simplest async arrangement with 1 task doing all IO for 1 computation component, which consists of 3 tasks. The computation component then writes a simple file with one variable, using distributed arrays.

Note that, due to the simplicity of this example the decomposition is created in the midst of the netCDF code that creates the file. In general the decomposition will be created before any files are created, and will be used for many files.

@edhartnett
Copy link
Contributor Author

I am going to take this PR down for a while.

@edhartnett edhartnett closed this Dec 24, 2017
@edhartnett edhartnett deleted the ejh_pio branch May 14, 2018 14:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants