Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

newer test class for passive elephant tests breaks parallelism in dask, in the case of brian2 backend. #230

Closed
russelljjarvis opened this issue Oct 27, 2019 · 4 comments

Comments

@russelljjarvis
Copy link
Contributor

russelljjarvis commented Oct 27, 2019

The new passive test class that replaces generate_prediction with get_result, and generate_features cannot be parallel mapped with dask, as it causes an error:
"cannot pickle weak reference".

The passive test class cannot be pickled using the pickle class either, for the same reason, I believe.

@russelljjarvis
Copy link
Contributor Author

This is only the case when the backend is brian2, and the model is adaptive exponential.

I think the reason is the passive tests try to store the model/simulator as a class attribute, and the brian2 module is assigned as a class attribute, but it is not picklable.

@russelljjarvis russelljjarvis changed the title newer test class for passive elephant tests breaks parallelism in dask. newer test class for passive elephant tests breaks parallelism in dask, in the case of brian2 backend. Oct 28, 2019
@rgerkin
Copy link
Contributor

rgerkin commented Oct 28, 2019

Which "passive test class" are you talking about?

Potential solution (though maybe there is something that should be done in the module itself): All SciUnit objects have an attribute unpicklable, which is a list. If you add the right attributes to that list (as strings with the names of the attributes), it will delete them (in a copy of the object) in the __getstate__ method (which pickle uses), which should then make it picklable. Of course, when unpickled, these attributes won't come automatically come back, but if the backend recreates them (as was the case with NEURONBackend, where initializing the simulation in each thread adds the h namespace back), then you should still be able to pickle and unpickle without a problem.

Don't worry about this right now.

@russelljjarvis
Copy link
Contributor Author

Yes, I found a work around. Brian2 adexp is currently the slowest backend however, so it will hinder long term performance

@russelljjarvis
Copy link
Contributor Author

russelljjarvis commented Nov 23, 2019

This was fixed. I mistakenly stored a brian2 model using self.model inside my divergent version of the passive.py test class file. Code is fast again now including passive tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants