Skip to content

Conversation

@Matthew-Whitlock
Copy link
Collaborator

A set of changes for:

  • A few minor bug fixes and memory management improvements
  • Adds function to query a list of failed processes
  • Replaces MPI overrides with a custom error handler
    • TODO: This breaks the MPI Comm store from Fenix, which removes our ability to automatically destroy old communicators. We can leave this as a user responsibility, or we can do partial MPI overrides for all comm creation functions.
  • Removes the Fenix request store, leaving it up to MPI to revoke any old MPI_Requests
  • Adds a few tests which use asynchronous functions

Matthew-Whitlock and others added 17 commits February 12, 2020 10:22
Fix some travis/testing issues.

Travis now pulls from ULFM master branch when it needs to rebuild ULFM.
Travis has an environment variable enabling oversubscription during the tests, instead of having that on all platforms when running make test
Tests that involve failure have their timeouts individually set to 1, so tests don't take 10+ seconds each w/ the default timeout of 10s
Simplified travis scripts (no more .travis_helpers directory)
New function, "Fenix_test_cancelled" for checking if pre-failure requests completed or were cancelled.

One thing to try finding a solution for: If a failure was found during an MPI_Test, that request has
already been removed from MPI internals and replaced w/ MPI_REQUEST_NULL. Fenix_test_cancelled will
report that this req was completed
This includes removing the option for comm_replace - users now must provide
a comm pointer to fenix_init and cannot rely on fenix to automatically replace
their input comm with the resilient comm.
@Matthew-Whitlock Matthew-Whitlock merged commit a6d6647 into sandialabs:master Nov 3, 2020
Matthew-Whitlock added a commit to Matthew-Whitlock/Fenix that referenced this pull request Apr 26, 2022
* Travis fixes (sandialabs#55)

Fix some travis/testing issues.

Travis now pulls from ULFM master branch when it needs to rebuild ULFM.
Travis has an environment variable enabling oversubscription during the tests, instead of having that on all platforms when running make test
Tests that involve failure have their timeouts individually set to 1, so tests don't take 10+ seconds each w/ the default timeout of 10s
Simplified travis scripts (no more .travis_helpers directory)

* Revert "Travis fixes (sandialabs#55)" (sandialabs#56)

Reverting un-reviewer PR, it was meant to be in my fork.

This reverts commit a41fd3b.

* Update README.md

* Merge updates for HCLIB (sandialabs#57)

* Add ability to query which processes failed

* Add support for MPI_Test

* Add support for testing pre-failure requests

* Fix bug when ERR_PROC_FAILED/ERR_REVOKED discovered in MPI_Test

* Fix MPI_Wait w/ cancelled requests

* Add missing file to commit

* Fix bug with MPI_STATUS_IGNORE

* Fix another bug with MPI_Test

* Add no-jump recovery option

* Travis fixes (#2)

Fix some travis/testing issues.

Travis now pulls from ULFM master branch when it needs to rebuild ULFM.
Travis has an environment variable enabling oversubscription during the tests, instead of having that on all platforms when running make test
Tests that involve failure have their timeouts individually set to 1, so tests don't take 10+ seconds each w/ the default timeout of 10s
Simplified travis scripts (no more .travis_helpers directory)

* First pass at removing the request store

New function, "Fenix_test_cancelled" for checking if pre-failure requests completed or were cancelled.

One thing to try finding a solution for: If a failure was found during an MPI_Test, that request has
already been removed from MPI internals and replaced w/ MPI_REQUEST_NULL. Fenix_test_cancelled will
report that this req was completed

* Implement custom errhandler

This includes removing the option for comm_replace - users now must provide
a comm pointer to fenix_init and cannot rely on fenix to automatically replace
their input comm with the resilient comm.

* Fenix comms are stack-allocated now, instead of malloced

* Cleanup redundant set_errhandler calls

* Fix data recovery bug

* Add usage instructions to all examples/tests

* Add support for MPI_Issend and MPI_Ssend (#3)

Merge in Issend test

Co-authored-by: mwhitlo@sandia.gov <mwhitlo@sandia.gov>
Co-authored-by: sriraj <srirajpaul@gmail.com>

Co-authored-by: Keita Teranishi <knteran@sandia.gov>
Co-authored-by: mwhitlo@sandia.gov <mwhitlo@sandia.gov>
Co-authored-by: sriraj <srirajpaul@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants