Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save results immediately #422

Open
wants to merge 5 commits into
base: develop
Choose a base branch
from

Conversation

PierreGtch
Copy link
Collaborator

Closes #419

Instead of waiting for the evaluation of a whole dataset to finish before saving the results, we could save them as soon as they are computed. Beneficial in case the evaluation fails in the middle.

Pending questions:

  • Can that create issues in case of parallel access to the hdf5 file?
  • What is the computational overhead due to accessing the hdf5 more often?

@PierreGtch PierreGtch marked this pull request as ready for review July 11, 2023 13:06
@PierreGtch
Copy link
Collaborator Author

@bruAristimunha the code change should be quite simple I think (see my commits) but I don't have time now to test the parallel case and benchmark in the standard case.

@bruAristimunha
Copy link
Collaborator

Okay, but now we lost all the results in the tutorials with this modification. We will need to debug this issue more to get it all green.

@bruAristimunha
Copy link
Collaborator

bruAristimunha commented Jul 11, 2023

I wonder if you could focus a little more on this problem in the CI, and then we could create a more specific test to work on the parallelization and overhead access issue. @PierreGtch

@PierreGtch
Copy link
Collaborator Author

I see it fails while testing the benchmark function. By default, benchmark uses -1 jobs. I think it might already be an issue with the parallel access to the file...

When I tested in local, the tests/evaluations.py were working fine

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Save results as soon as they are computed
3 participants