Rewrite create_synthetic_data

[ChristianZimpelmann]

What problem do you want to solve?

create_synthetic_data is currently overly complicated and slow (especially when many households are created at the same time). Main problem seems to be that a lot of pandas objects are created and concatenated.

To fix the problems, I rewrote most of the code. Remarks:

• all variables that do not vary across households are now collected in a dict of lists: specs_constant_over_households
• all variables that should vary across households are now collected in a dict of lists of lists: specs_heterogeneous
• At the moment, it is no longer possible to create single and couple households or households with varying numbers of children with one function call. I would deliberately drop this feature, but this is up for discussion and could be implemented, as well.

Todo

• Pick an appropriate title.
• Rewrite code.
• Improve documentation.
• Adjust test cases
• Adjust tutorials
• Document PR in CHANGES.md.

Collection of ideas

• allow for more than 2 children by either extending official data on Bedarfsgemeinschaften or by dropping use of official data
• allow for easier specification of variables which are constant within households via broadcasting (i.e. allow for mietstufe: 2 instead of mietstufe: [2, 2, 2, 2]

[hmgaudecker]

Thanks, sounds great!

+1 for removing that feature

[hmgaudecker]

Any chance to get this merged within a couple of days?

[ChristianZimpelmann]

I can have a look at it on Monday.

[codecov]

Codecov Report

Patch coverage: 98.24% and project coverage change: -0.13 ⚠️

Comparison is base (993fc26) 94.20% compared to head (a4b870a) 94.07%.

❗ Current head a4b870a differs from pull request most recent head 51bd1c1. Consider uploading reports for the commit 51bd1c1 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #514      +/-   ##
==========================================
- Coverage   94.20%   94.07%   -0.13%     
==========================================
  Files          48       48              
  Lines        2983     2903      -80     
==========================================
- Hits         2810     2731      -79     
+ Misses        173      172       -1     

Impacted Files	Coverage Δ
src/_gettsim/synthetic.py	98.46% <98.24%> (+3.28%)	⬆️

... and 3 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[ChristianZimpelmann]

• I adjusted and expanded test cases and solved the merge conflict.
• I would add the "Collection of ideas" to a new issues
• only open ToDo is the adjustments of tutorials. I would only address this after we fixed BUG: Important print statements have been deleted in tutorials #570.

[ChristianZimpelmann]

Somehow, the last commit 4bd9432a4aaaa8538367fef9dbaa3f01b74cb0f9 shows up on the branch, but not in this PR yet.

@hmgaudecker, please check again before your review.

[hmgaudecker]

Very nice, much clearer than before! Just a smaller things.

[ChristianZimpelmann]

Thanks for the feedback! I addressed all the comments now.

[hmgaudecker]

Excellent, thank you!

[hmgaudecker]

Only realised ex post that docs are failing because some notebook seems to be using the old interface of create_synthetic_data, please fix that before merging, should be trivial.

[ChristianZimpelmann]

Thanks!

See my comment above:

• only open ToDo is the adjustments of tutorials. I would only address this after we fixed BUG: Important print statements have been deleted in tutorials #570.

The documentation is currently incomplete. Easiest to fix is probably to revert to the state before the change of pre-commit hooks. So we shouldn't make any changes to the current version on main.

So I think the options are:

• merge without adjusting tutorials and keep an issue linked to BUG: Important print statements have been deleted in tutorials #570
• fix BUG: Important print statements have been deleted in tutorials #570 first

[hmgaudecker]

Hope that last commit will do, this brought all print-statements back in and changed the hooks.

[ChristianZimpelmann]

Thanks for fixing the tutorials! I updated them now using the new interface of create_synthetic

[hmgaudecker]

Excellent, thanks! Just a couple of typos / language details, will implement myself.