Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change main delay calculation from simple for-loop to parallel for-loop #32

Closed
jlmaurer opened this issue Apr 27, 2020 · 1 comment
Closed

Comments

@jlmaurer
Copy link
Collaborator

jlmaurer commented Apr 27, 2020

Code is tools/RAiDER/delayFcns.py:

with h5py.File(pnts_file, 'r') as f:

The code is chunked but it processes each chunk serially. A potentially major speed-up would be to add a parallel implementation here. Only reads are required; no data is written, but I'm not sure what happens if you try to open an HDF5 file in read-only mode (see perhaps here for a start). My hope is that we can just pass the file name to different processes (and if the file is chunked properly so we never try to read the same chunk) we can just use multiprocessing and do it that way. I.e. something like:

zipped_args = zip([filename]*len(other_args), other_arg_1, other_arg_2,...)
for arg_set in zipped_args:
    open the file in read-only mode and read the appropriate chunk
    do the delay calculation

But I don't know if HDF5 will actually allow that to work.

@jlmaurer
Copy link
Collaborator Author

PR #45 addresses this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants