Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stage-in/stage-out + major adaptor refactor #453

Merged
merged 66 commits into from
Oct 13, 2022
Merged

Stage-in/stage-out + major adaptor refactor #453

merged 66 commits into from
Oct 13, 2022

Commits on Sep 19, 2022

  1. Interceptor API class

    @lukemartinlogan
    lukemartinlogan committed Sep 19, 2022
    Configuration menu
    Copy the full SHA
    19112fb View commit details
    Browse the repository at this point in the history

  2. Fix some lint issues

    @lukemartinlogan
    lukemartinlogan committed Sep 19, 2022
    Configuration menu
    Copy the full SHA
    59cc41e View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2022

  1. Posix interceptor compiles

    @lukemartinlogan
    lukemartinlogan committed Sep 20, 2022
    Configuration menu
    Copy the full SHA
    c783b2d View commit details
    Browse the repository at this point in the history

  2. Add different read cases

    @lukemartinlogan
    lukemartinlogan committed Sep 20, 2022
    Configuration menu
    Copy the full SHA
    8cb4b86 View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2022

  1. Posix refactor passes basic tests

    @lukemartinlogan
    lukemartinlogan committed Sep 21, 2022
    Configuration menu
    Copy the full SHA
    b269345 View commit details
    Browse the repository at this point in the history

  2. Fix lint issues. Remove posix save.

    @lukemartinlogan
    lukemartinlogan committed Sep 21, 2022
    Configuration menu
    Copy the full SHA
    d5601fe View commit details
    Browse the repository at this point in the history

  3. Add singleton test

    @lukemartinlogan
    lukemartinlogan committed Sep 21, 2022
    Configuration menu
    Copy the full SHA
    03f1ae1 View commit details
    Browse the repository at this point in the history

  4. Move file_into after is_tracked to avoid memory corruption in fxstat … 

    …intercept. Make singleton return C ptr instead of shared_ptr.
    @lukemartinlogan
    lukemartinlogan committed Sep 21, 2022
    Configuration menu
    Copy the full SHA
    90117b1 View commit details
    Browse the repository at this point in the history

  5. Remove posix_test_singleton

    @lukemartinlogan
    lukemartinlogan committed Sep 21, 2022
    Configuration menu
    Copy the full SHA
    865391a View commit details
    Browse the repository at this point in the history

  6. Remove the pool variable after GetInstance in cases where it is not used

    @lukemartinlogan
    lukemartinlogan committed Sep 21, 2022
    Configuration menu
    Copy the full SHA
    94a52d5 View commit details
    Browse the repository at this point in the history

Commits on Sep 22, 2022

  1. Use BlobPlacementIter instead of passing all params over and over

    @lukemartinlogan
    lukemartinlogan committed Sep 22, 2022
    Configuration menu
    Copy the full SHA
    6ea28a0 View commit details
    Browse the repository at this point in the history

  2. NonType PlacementPolicy

    @lukemartinlogan
    lukemartinlogan committed Sep 22, 2022
    Configuration menu
    Copy the full SHA
    72b0765 View commit details
    Browse the repository at this point in the history

  3. Add sync and close to hermes fs

    @lukemartinlogan
    lukemartinlogan committed Sep 22, 2022
    Configuration menu
    Copy the full SHA
    c066437 View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2022

  1. Begin refactoring stdio adapter. Change the method for generating ada… 

    …pter real api classes.
    @lukemartinlogan
    lukemartinlogan committed Sep 23, 2022
    Configuration menu
    Copy the full SHA
    e5b4b8f View commit details
    Browse the repository at this point in the history

  2. Change nomenclature of internal adapter files to use fs_api and real_api

    @lukemartinlogan
    lukemartinlogan committed Sep 23, 2022
    Configuration menu
    Copy the full SHA
    d32e199 View commit details
    Browse the repository at this point in the history

  3. Single-thread stdio tests pass

    @lukemartinlogan
    lukemartinlogan committed Sep 23, 2022
    Configuration menu
    Copy the full SHA
    84380f6 View commit details
    Browse the repository at this point in the history

  4. Fix lint issues

    @lukemartinlogan
    lukemartinlogan committed Sep 23, 2022
    Configuration menu
    Copy the full SHA
    e18c040 View commit details
    Browse the repository at this point in the history

  5. Fix unused parameter f in Tell()

    @lukemartinlogan
    lukemartinlogan committed Sep 23, 2022
    Configuration menu
    Copy the full SHA
    d9aabee View commit details
    Browse the repository at this point in the history

  6. Uncomment fs_api in freopen

    @lukemartinlogan
    lukemartinlogan committed Sep 23, 2022
    Configuration menu
    Copy the full SHA
    140f9a7 View commit details
    Browse the repository at this point in the history

  7. Properly comment fs_api out in freopen

    @lukemartinlogan
    lukemartinlogan committed Sep 23, 2022
    Configuration menu
    Copy the full SHA
    0981644 View commit details
    Browse the repository at this point in the history

  8. Remove unnecessary mdm variables from mapper tests

    @lukemartinlogan
    lukemartinlogan committed Sep 23, 2022
    Configuration menu
    Copy the full SHA
    163b250 View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2022

  1. Make reopen_internal use ret instead of stream. Exit in bkt->Destroy.

    @lukemartinlogan
    lukemartinlogan committed Sep 25, 2022
    Configuration menu
    Copy the full SHA
    031a2d7 View commit details
    Browse the repository at this point in the history

  2. Two files are equal if their inode and dev are equal.

    @lukemartinlogan
    lukemartinlogan committed Sep 25, 2022
    Configuration menu
    Copy the full SHA
    3c23982 View commit details
    Browse the repository at this point in the history

  3. Lint issues

    @lukemartinlogan
    lukemartinlogan committed Sep 25, 2022
    Configuration menu
    Copy the full SHA
    712d854 View commit details
    Browse the repository at this point in the history

  4. Remove some comments

    @lukemartinlogan
    lukemartinlogan committed Sep 25, 2022
    Configuration menu
    Copy the full SHA
    d0ffa1e View commit details
    Browse the repository at this point in the history

  5. Simplify namespace for Singleton. Change PersistEagerly -> IsAsyncFlush.

    @lukemartinlogan
    lukemartinlogan committed Sep 25, 2022
    Configuration menu
    Copy the full SHA
    66f7aa3 View commit details
    Browse the repository at this point in the history

  6. Make it so that the RAM tier has more space for the DPE test to accou… 

    …nt for Arena.
    @lukemartinlogan
    lukemartinlogan committed Sep 25, 2022
    Configuration menu
    Copy the full SHA
    6709996 View commit details
    Browse the repository at this point in the history

  7. Start MPI-IO interceptor refactor

    @lukemartinlogan
    lukemartinlogan committed Sep 25, 2022
    Configuration menu
    Copy the full SHA
    3c0f4bf View commit details
    Browse the repository at this point in the history

  8. Simplify fgetpos in stdio

    @lukemartinlogan
    lukemartinlogan committed Sep 25, 2022
    Configuration menu
    Copy the full SHA
    c51e274 View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2022

  1. MPI interceptor generator shouldn't have #includes.

    @lukemartinlogan
    lukemartinlogan committed Sep 26, 2022
    Configuration menu
    Copy the full SHA
    82d7859 View commit details
    Browse the repository at this point in the history

  2. Stage in compiles

    @lukemartinlogan
    lukemartinlogan committed Sep 26, 2022
    Configuration menu
    Copy the full SHA
    891b0a0 View commit details
    Browse the repository at this point in the history

  3. use real_api instead of _real_ syntax for mpiio

    @lukemartinlogan
    lukemartinlogan committed Sep 26, 2022
    Configuration menu
    Copy the full SHA
    86841e9 View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2022

  1. Utilize DPE option in reads to an extent

    @lukemartinlogan
    lukemartinlogan committed Sep 27, 2022
    Configuration menu
    Copy the full SHA
    f5736b8 View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2022

  1. IoOptions instead of many params. CoordinatedPut.

    @lukemartinlogan
    lukemartinlogan committed Sep 30, 2022
    Configuration menu
    Copy the full SHA
    837ae63 View commit details
    Browse the repository at this point in the history

  2. Add an error check for CoordinatedPut.

    @lukemartinlogan
    lukemartinlogan committed Sep 30, 2022
    Configuration menu
    Copy the full SHA
    5b5424e View commit details
    Browse the repository at this point in the history

  3. Remove std::move

    @lukemartinlogan
    lukemartinlogan committed Sep 30, 2022
    Configuration menu
    Copy the full SHA
    970bba3 View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2022

  1. Use Write() instead of Put

    @lukemartinlogan
    lukemartinlogan committed Oct 3, 2022
    Configuration menu
    Copy the full SHA
    86629e3 View commit details
    Browse the repository at this point in the history

  2. Ticket mutex init

    @lukemartinlogan
    lukemartinlogan committed Oct 3, 2022
    Configuration menu
    Copy the full SHA
    2a700a3 View commit details
    Browse the repository at this point in the history

Commits on Oct 5, 2022

  1. Adapter refactor at least compiles

    @lukemartinlogan
    lukemartinlogan committed Oct 5, 2022
    Configuration menu
    Copy the full SHA
    8601482 View commit details
    Browse the repository at this point in the history

  2. Identified issue with GetFilenameFromFP for MPI

    @lukemartinlogan
    lukemartinlogan committed Oct 5, 2022
    Configuration menu
    Copy the full SHA
    4564b3e View commit details
    Browse the repository at this point in the history

  3. MPI adapter refactor passes tests

    @lukemartinlogan
    lukemartinlogan committed Oct 5, 2022
    Configuration menu
    Copy the full SHA
    93a4a89 View commit details
    Browse the repository at this point in the history

  4. Fix lint issues

    @lukemartinlogan
    lukemartinlogan committed Oct 5, 2022
    Configuration menu
    Copy the full SHA
    44d7940 View commit details
    Browse the repository at this point in the history

  5. Fix Ticket constructor order

    @lukemartinlogan
    lukemartinlogan committed Oct 5, 2022
    Configuration menu
    Copy the full SHA
    5622c81 View commit details
    Browse the repository at this point in the history

  6. Make LockBlob and UnlockBlob use pid and tid

    @lukemartinlogan
    lukemartinlogan committed Oct 5, 2022
    Configuration menu
    Copy the full SHA
    524e659 View commit details
    Browse the repository at this point in the history

  7. Reorder constructors for AdapterStat and File

    @lukemartinlogan
    lukemartinlogan committed Oct 5, 2022
    Configuration menu
    Copy the full SHA
    4f428dc View commit details
    Browse the repository at this point in the history

  8. Fix implicit data type conversions in mpiio fs_api

    @lukemartinlogan
    lukemartinlogan committed Oct 5, 2022
    Configuration menu
    Copy the full SHA
    049e1ff View commit details
    Browse the repository at this point in the history

  9. Fix implicit data type conversions in mpiio fs_api

    @lukemartinlogan
    lukemartinlogan committed Oct 5, 2022
    Configuration menu
    Copy the full SHA
    8a4776e View commit details
    Browse the repository at this point in the history

  10. Remove unused variables in interceptor

    @lukemartinlogan
    lukemartinlogan committed Oct 5, 2022
    Configuration menu
    Copy the full SHA
    490740b View commit details
    Browse the repository at this point in the history

Commits on Oct 6, 2022

  1. Delete hermes req

    @lukemartinlogan
    lukemartinlogan committed Oct 6, 2022
    Configuration menu
    Copy the full SHA
    c3f034f View commit details
    Browse the repository at this point in the history

  2. Add IoStatus to I/O functions

    @lukemartinlogan
    lukemartinlogan committed Oct 6, 2022
    Configuration menu
    Copy the full SHA
    4a5e0d8 View commit details
    Browse the repository at this point in the history

  3. Create _IoStats callback for every I/O operation.

    @lukemartinlogan
    lukemartinlogan committed Oct 6, 2022
    Configuration menu
    Copy the full SHA
    70da834 View commit details
    Browse the repository at this point in the history

  4. Use hreq instead of req

    @lukemartinlogan
    lukemartinlogan committed Oct 6, 2022
    Configuration menu
    Copy the full SHA
    34559eb View commit details
    Browse the repository at this point in the history

  5. Add more info to log for (un)coordinated put

    @lukemartinlogan
    lukemartinlogan committed Oct 6, 2022
    Configuration menu
    Copy the full SHA
    ed7c7c9 View commit details
    Browse the repository at this point in the history

  6. Most MPI-IO test cases pass with concurrency

    @lukemartinlogan
    lukemartinlogan committed Oct 6, 2022
    Configuration menu
    Copy the full SHA
    21ae93e View commit details
    Browse the repository at this point in the history

  7. AWriteOrdered

    @lukemartinlogan
    lukemartinlogan committed Oct 6, 2022
    Configuration menu
    Copy the full SHA
    dc52a97 View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2022

  1. iWriteShared and WriteShared

    @lukemartinlogan
    lukemartinlogan committed Oct 7, 2022
    Configuration menu
    Copy the full SHA
    1e7f94a View commit details
    Browse the repository at this point in the history

  2. Append rank to vbucket in Sync

    @lukemartinlogan
    lukemartinlogan committed Oct 7, 2022
    Configuration menu
    Copy the full SHA
    fba23c0 View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2022

  1. Mpiio async write ordered should set hreq->io_status

    @lukemartinlogan
    lukemartinlogan committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    5b6c452 View commit details
    Browse the repository at this point in the history

  2. Ensure that amode is 0

    @lukemartinlogan
    lukemartinlogan committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    2246ef6 View commit details
    Browse the repository at this point in the history

  3. Use MPI_COMM_SELF instead of COMM_WORLD for stdio and posix MPI

    @lukemartinlogan
    lukemartinlogan committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    305ae9a View commit details
    Browse the repository at this point in the history

Commits on Oct 13, 2022

  1. Add a specific stager unit test for file stage-in

    @lukemartinlogan
    lukemartinlogan committed Oct 13, 2022
    Configuration menu
    Copy the full SHA
    a842449 View commit details
    Browse the repository at this point in the history

  2. Use vectors in stage-in test instead of malloc

    @lukemartinlogan
    lukemartinlogan committed Oct 13, 2022
    Configuration menu
    Copy the full SHA
    cc983de View commit details
    Browse the repository at this point in the history

  3. Fix lint issues

    @lukemartinlogan
    lukemartinlogan committed Oct 13, 2022
    Configuration menu
    Copy the full SHA
    b5b1f83 View commit details
    Browse the repository at this point in the history

  4. Merge documentation. Add vbucket.h and set to filesystem.h

    @lukemartinlogan
    lukemartinlogan committed Oct 13, 2022
    Configuration menu
    Copy the full SHA
    1dab3e4 View commit details
    Browse the repository at this point in the history

  5. Remove datastructures.h

    @lukemartinlogan
    lukemartinlogan committed Oct 13, 2022
    Configuration menu
    Copy the full SHA
    6019f85 View commit details
    Browse the repository at this point in the history

  6. git push origin stage-in 

    Merge branch 'HDFGroup-master' into stage-in
    @lukemartinlogan
    lukemartinlogan committed Oct 13, 2022
    Configuration menu
    Copy the full SHA
    f4e4fc8 View commit details
    Browse the repository at this point in the history