Numpyrize backward induction.

[tobiasraabe]

Problem

One major bottleneck of the Python implementation is in pyth_backward_induction which accounts for the majority of time used. Even for a small state space (10 periods, 2 types) extracting a state and finding the emax of its children states in the subsequent period takes 11% and 86% of the time. This obstacle can only be alleviated if the procedure can be translated to Numpy and Numba.

Solution

The final solution is very fast. Getting the emax from subsequent periods is no obstacle anymore and will not be a bottleneck anytime soon. The new bottleneck which accounts for over 92% of the time is construct_emax_risk. You can find the profiling results for 10 evaluations in the appendix.

• Rewrite the state space to contain only Numpy arrays.
• Include pyth_create_systematic_rewards in state space and add an update function for the rewards used in pyth_criterion.
• Numpyrize complete pyth_backward_induction
• Adjust pyth_simulate, pyth_contributions and other related functions to the new state space.
• Fix test suite and reimplement skipped tests
• Check 50 regression tests
• Test explicit parallelization in get_emaxs_sub, but it doubles runtime.
• Rework construct_emax_risk as independent gufunc.
• Added another restriction to pyth_create_state_space so that no states are allowed were an agent chose always occupation A or B or education and lagged choice is home. (Thanks @mo2561057)
• testing_pull_request.py works.

Appendix

• profile_pyth_backward_induction.txt

[tobiasraabe]

Hi @janosg, I am not able to reproduce the error in test_f2py::test_6 on my machine. Check out the second failed run on travis as the first log is not very informative. Can you give me a hint?

[janosg]

@tobiasraabe, On my machine test_f2py::test_6 also passes.

[tobiasraabe]

Did you also not deactivate mpi or omp? Now, if I deactivate one of them during compilation, I get a segementation fault.

[janosg]

If I deactivate any of the two or both I also get a segfault with no further information that would help you.

[tobiasraabe]

Thanks for helping me! The seg fault was not caused by Fortran, but by an indexerror in a jitted function which also returns a segmentation fault. I thought Numba is more verbose than this.

[janosg]

No, in general numba segfaults are much harder to diagnose than fortran segfaults!

[peisenha]

Thanks for the great work! I create some issues to address later when we have settled down in our development a bit after these tumultous times :)

[tobiasraabe]

I cancelled the build as it took too much time. But, tests were successful on my machine, the Python suite was fine on Appveyor and Fortran on Travis.