how to run dso multiple times in a loop

[joshkamm]

Are there any special considerations that need to be taken into account when using DeepSymbolicOptimizer multiple times within a python program?

Currently we are looping over a set of datasets. In each loop iteration we create a new DeepSymbolicOptimizer object and execute training on it. This seems to work when running on a single core other than of course being slow. When running on multiple cores the program seems to deadlock. Specifically, all printing stops and running top on linux reveals all python processes all using 0.0% CPU. When we do a keyboard interrupt, many processes return stack traces that seem to be waiting for locks. If it's helpful we can provide more detail on the code and or the specific stack traces.

[brendenpetersen]

That's a good question. In general the DeepSymbolicOptimizer object is not thread safe--tensorflow does not make creating multiple TF-based objects like this easy.

One option is to use the n_cores_batch parameter. Your multiple training runs will still run in series, but each one will leverage multiple cores.

Otherwise, for batch tasks, you'll be better off parallelizing the entire program and having each one create a single DeepSymbolicOptimizer object called with your different datasets.

[joshkamm]

Okay thank you for the explanation.

We tried parallelizing the entire program and restricting to one DeepSymbolicOptimizer object per process and that seems to be working.