Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stage-in/stage-out + major adaptor refactor #453

Merged
merged 66 commits into from
Oct 13, 2022
Merged

Stage-in/stage-out + major adaptor refactor #453

merged 66 commits into from
Oct 13, 2022

Conversation

lukemartinlogan
Copy link
Collaborator

This PR adds the functionality of stage-in and stage-out, and a re-invisioning of the POSIX, STDIO, and MPI-IO adapter codes.

This PR addresses the following issues:

  1. Implement stage in/out #444: Provided a basic implementation of stage-in / stage-out for UNIX files. Includes directory and file stage-in.
  2. Diagnose sporadically failing MPIIO adapter test #413: Numerous concurrency issues in existing MPI adapter code. Added Lock to fix.
  3. Investigate failing MPI/IO adapter test #337: Numerous concurrency issues in existing MPI adapter code. Added Lock to fix.
  4. Refactor adapters to share and reuse common code #285: Made it so that POSIX, MPI-IO, and STDIO inherit the same filesystem base class code. Reduces errors among the different codes.
  5. Implement asynchronous flushing in POSIX adapter #250: STDIO and POSIX inherit same code, including async I/O functionality
  6. Diagnose failing MPI-IO adapter test #209: Numerous concurrency issues in existing MPI adapter code. Added Lock to fix.
  7. Need a better way to distinguish daemon vs. client in adapters #206: Use export HERMES_CLIENT=1 to specify that it is in client mode, not daemon.

@lukemartinlogan
Interceptor API class
19112fb
@lukemartinlogan
Fix some lint issues
59cc41e
@lukemartinlogan
Posix interceptor compiles
c783b2d
@lukemartinlogan
Add different read cases
8cb4b86
@lukemartinlogan
Posix refactor passes basic tests
b269345
@lukemartinlogan
Fix lint issues. Remove posix save.
d5601fe
@lukemartinlogan
Add singleton test
03f1ae1
@lukemartinlogan
Move file_into after is_tracked to avoid memory corruption in fxstat …
90117b1 
…intercept. Make singleton return C ptr instead of shared_ptr.
@lukemartinlogan
Remove posix_test_singleton
865391a
@lukemartinlogan
Remove the pool variable after GetInstance in cases where it is not used
94a52d5
@lukemartinlogan
Use BlobPlacementIter instead of passing all params over and over
6ea28a0
@lukemartinlogan
NonType PlacementPolicy
72b0765
@lukemartinlogan
Add sync and close to hermes fs
c066437
@lukemartinlogan
Begin refactoring stdio adapter. Change the method for generating ada…
e5b4b8f 
…pter real api classes.
@lukemartinlogan
Change nomenclature of internal adapter files to use fs_api and real_api
d32e199
@lukemartinlogan
Single-thread stdio tests pass
84380f6
@lukemartinlogan
Fix lint issues
e18c040
@lukemartinlogan
Fix unused parameter f in Tell()
d9aabee
@lukemartinlogan
Uncomment fs_api in freopen
140f9a7
@lukemartinlogan
Properly comment fs_api out in freopen
0981644
@lukemartinlogan
Remove unnecessary mdm variables from mapper tests
163b250
@lukemartinlogan
Make reopen_internal use ret instead of stream. Exit in bkt->Destroy.
031a2d7
@lukemartinlogan
Two files are equal if their inode and dev are equal.
3c23982
@lukemartinlogan
Lint issues
712d854
@lukemartinlogan
Remove some comments
d0ffa1e
@lukemartinlogan
Simplify namespace for Singleton. Change PersistEagerly -> IsAsyncFlush.
66f7aa3
@lukemartinlogan
Make it so that the RAM tier has more space for the DPE test to accou…
6709996 
…nt for Arena.
@lukemartinlogan
Start MPI-IO interceptor refactor
3c0f4bf
@lukemartinlogan
Simplify fgetpos in stdio
c51e274
@lukemartinlogan
MPI interceptor generator shouldn't have #includes.
82d7859
@lukemartinlogan
Adapter refactor at least compiles
8601482
@lukemartinlogan
Identified issue with GetFilenameFromFP for MPI
4564b3e
@lukemartinlogan
MPI adapter refactor passes tests
93a4a89
@lukemartinlogan
Fix lint issues
44d7940
@lukemartinlogan
Fix Ticket constructor order
5622c81
@lukemartinlogan
Make LockBlob and UnlockBlob use pid and tid
524e659
@lukemartinlogan
Reorder constructors for AdapterStat and File
4f428dc
@lukemartinlogan
Fix implicit data type conversions in mpiio fs_api
049e1ff
@lukemartinlogan
Fix implicit data type conversions in mpiio fs_api
8a4776e
@lukemartinlogan
Remove unused variables in interceptor
490740b
@lukemartinlogan
Delete hermes req
c3f034f
@lukemartinlogan
Add IoStatus to I/O functions
4a5e0d8
@lukemartinlogan
Create _IoStats callback for every I/O operation.
70da834
@lukemartinlogan
Use hreq instead of req
34559eb
@lukemartinlogan
Add more info to log for (un)coordinated put
ed7c7c9
@lukemartinlogan
Most MPI-IO test cases pass with concurrency
21ae93e
@lukemartinlogan
AWriteOrdered
dc52a97
@lukemartinlogan
iWriteShared and WriteShared
1e7f94a
@lukemartinlogan
Append rank to vbucket in Sync
fba23c0
@lukemartinlogan
Mpiio async write ordered should set hreq->io_status
5b6c452
@lukemartinlogan
Ensure that amode is 0
2246ef6
@lukemartinlogan
Use MPI_COMM_SELF instead of COMM_WORLD for stdio and posix MPI
305ae9a
@lukemartinlogan
Add a specific stager unit test for file stage-in
a842449
@lukemartinlogan
Use vectors in stage-in test instead of malloc
cc983de
@lukemartinlogan
Fix lint issues
b5b1f83
@lukemartinlogan
Merge documentation. Add vbucket.h and set to filesystem.h
1dab3e4
@lukemartinlogan
Remove datastructures.h
6019f85
@lukemartinlogan
git push origin stage-in
f4e4fc8 
Merge branch 'HDFGroup-master' into stage-in
@lukemartinlogan lukemartinlogan changed the title Stag-in/stage-out + major adaptor refactor Stage-in/stage-out + major adaptor refactor Oct 13, 2022
@lukemartinlogan lukemartinlogan merged commit 88cc715 into HDFGroup:master Oct 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@lukemartinlogan