Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallel reading #23

Open
JanVogelsang opened this issue Feb 2, 2021 · 2 comments
Open

Parallel reading #23

JanVogelsang opened this issue Feb 2, 2021 · 2 comments
Labels
enhancement New feature or request help wanted Extra attention is needed wontfix This will not be worked on
Projects

Comments

@JanVogelsang
Copy link
Collaborator

It should be possible to read in data for most classes using multiple threads or even processes (processes might use another data bus and therefore be more performant, will have to check if that really is the case).

@JanVogelsang JanVogelsang self-assigned this Feb 2, 2021
@JanVogelsang JanVogelsang added the enhancement New feature or request label Feb 2, 2021
@JanVogelsang JanVogelsang added this to the Beta milestone Feb 2, 2021
@JanVogelsang JanVogelsang added this to Utils in Roadmap Feb 2, 2021
@JanVogelsang JanVogelsang removed their assignment Feb 18, 2021
@JanVogelsang JanVogelsang modified the milestones: Beta, Alpha, Release Feb 18, 2021
@JanVogelsang JanVogelsang added help wanted Extra attention is needed wontfix This will not be worked on labels Feb 18, 2021
@JanVogelsang
Copy link
Collaborator Author

For all cases that has been tested, read times have not been any problem.

If you experience enormously long loading times, please send your FDS case and we might think of further improving the reading performance.

@JanVogelsang
Copy link
Collaborator Author

Some simple tests didn't yield any noticeable speedup when reading in data using multiple threads. For small amounts of data (~30MB files), the time it took to read in and process all data even increased by about 5%.

For most use cases simple parallelization won't be applicable and more sophisticated approaches are needed. For HPC systems various approaches will yield great results, however the same approaches will most likely not work for common PC-systems due to only having a single memory and disk bus.

@JanVogelsang JanVogelsang removed this from the Release milestone May 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is needed wontfix This will not be worked on
Projects
Roadmap
Utils
Development

No branches or pull requests

1 participant