Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

shared cache: add shared-file support to AXL to parallelize copy #489

Open
mcfadden8 opened this issue Mar 4, 2022 · 0 comments
Open

shared cache: add shared-file support to AXL to parallelize copy #489

mcfadden8 opened this issue Mar 4, 2022 · 0 comments
Projects

Comments

@mcfadden8
Copy link
Collaborator

From Adam:
With the current version of AXL, we'd have a single thread on a single compute node copy the entire file from cache to the parallel file system. However, since we're now dealing with a single shared file, the size of the file scales up with the node count of the job. We probably will want to split up a file into regions and assign different portions of the file to be copied by different (multiple) processes. And that seems like it will require us to modify the AXL API so that we can provide an offset and length in addition to the source and destination file names.

@mcfadden8 mcfadden8 created this issue from a note in v3.1 (To do) Mar 4, 2022
@mcfadden8 mcfadden8 moved this from To do to In progress in v3.1 Mar 4, 2022
@adammoody adammoody changed the title Add shared-file cache support to AXL shared cache: add shared-file support to AXL to parallelize copy Mar 5, 2022
This was referenced Apr 1, 2022
@mcfadden8 mcfadden8 mentioned this issue May 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
v3.1
In progress
Development

No branches or pull requests

1 participant