Limit number of parallel processes per task

[ClaraBuettner]

Closes #265

[ClaraBuettner]

@gnn: The number of parallel processes is currently manually set to one in each tasks which uses multi processing. I could also add a variable, e.g. max_processes_per_task to the configuration file which is one by default but could also be increased.

[ClaraBuettner]

The number of parallel process per task is now part of the cli and stored in the egon-data.configuration file. It is just one value for all tests, it's not task-specific because this would require information about the task dependencies.
I tested it and it worked fine.

[ClaraBuettner]

I updated the branch with the current dev and run it in test mode.

@nesnoj The task hh_demand_profiles.houseprofiles-in-census-cells takes 6:30 min with processes-per-task=2 and 8 min with processes-per-task=1 for Schleswig Holstein. So it is slower, but not linear to the number of processes.
If you're ok with this, I will merge this into the continuous integration branch to see how it performs for Germany with processes-per-task=2.

[nesnoj]

If you're ok with this, I will merge this into the continuous integration branch to see how it performs for Germany with processes-per-task=2.

Yepp, go ahead 👍

[ClaraBuettner]

This branch was somehow merged intodev via another branch but the changes are not on dev anymore. I'm wondering why this happened...
@KathiEsterl: Was it intended to merge this branch?

[KathiEsterl]

This branch was somehow merged intodev via another branch but the changes are not on dev anymore. I'm wondering why this happened... @KathiEsterl: Was it intended to merge this branch?

I never meant to do anything with this branch!
I think while commiting something to the DSM-branch I used the Hashtag for this PR although I probably meant something else... That's how this got mixed up. It looks like I merged it together with the DSM-branch then, but I am not sure as I had checked all the changes in the DSM-branch before merging in detail. I don't really know what happened here...
I am sorry for that! Let me know if I can help with that.

[ClaraBuettner]

I never meant to do anything with this branch! I think while commiting something to the DSM-branch I used the Hashtag for this PR although I probably meant something else... That's how this got mixed up. It looks like I merged it together with the DSM-branch then, but I am not sure as I had checked all the changes in the DSM-branch before merging in detail. I don't really know what happened here... I am sorry for that! Let me know if I can help with that.

I can imagine that it happened because there was still the commit in your branch, even if it was reverted later git thought this was merged. The code changes are not in dev branch, if we want to have them I can re-do the commits.
I already checked that this only happened in this PR.