Include newborns' initial conditions in make_shock_history and read_shocks==True

[Mv77]

Fixes #1097.

make_shock_history now creates an additional agent property called newborn_init_history, a dictionary of T_sim x AgentCount arrays, one for each state variable. The arrays are full of nans except for times, agents and states that are set when an agent is reborn.

For example, if agent 35 is re-born at time 7, then newborn_init_history['aNrm'][7,35] will be the aNrm value that has to be used to initialize the re-born agent.

[Mv77]

An annoying aspect of this solution is that I have to have nan arrays for all state variables, not only those that are used to initialize agents. For instance, IndShockConsumerType agents require only [aNrm, pLvl] to be initialized, but they have many other states (mNrm, PAggLvl, bNrm...).

HARK currently does not declare which subset of the state variables are required to initialize an agent, and that is why my general solution creates arrays for all states. An alternative would be to give AgentType a new attribute called something like newborn_vars: a list of strings indicating which of the state variables are drawn by sim_births. I'd like to get people's thoughts on whether this would be a better solution.

[codecov-commenter]

Codecov Report

Merging #1101 (8197ea0) into master (36b5d1c) will increase coverage by 0.12%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #1101      +/-   ##
==========================================
+ Coverage   73.73%   73.85%   +0.12%     
==========================================
  Files          69       69              
  Lines       10579    10631      +52     
==========================================
+ Hits         7800     7852      +52     
  Misses       2779     2779              

Impacted Files	Coverage Δ
...nsumptionSaving/tests/test_IndShockConsumerType.py	75.08% <ø> (+2.00%)	⬆️
HARK/core.py	90.40% <100.00%> (+0.63%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 36b5d1c...8197ea0. Read the comment docs.

[sbenthall]

I see how this is connected to the issue in #1099

I suppose 'post_states' used to be those states that required newborn initialization.

Assuming that k_t is used, then I suppose we can remove post_states entirely.

I believe that the state variables that "must" be drawn/filled for newborns can be identified from from dependency graph implied by the equations.

[Mv77]

Yes,

I believe the variables would be those needed by get_states:

HARK/HARK/core.py

Lines 541 to 559 in 9562caf

	self.get_mortality() # Replace some agents with "newborns"
	# state_{t-1}
	for var in self.state_now:
	self.state_prev[var] = self.state_now[var]
	if isinstance(self.state_now[var], np.ndarray):
	self.state_now[var] = np.empty(self.AgentCount)
	else:
	# Probably an aggregate variable. It may be getting set by the Market.
	pass
	if self.read_shocks: # If shock histories have been pre-specified, use those
	self.read_shocks_from_history()
	else: # Otherwise, draw shocks as usual according to subclass-specific method
	self.get_shocks()
	self.get_states() # Determine each agent's state at decision time
	self.get_controls() # Determine each agent's choice or control variables based on states
	self.get_poststates() # Move now state_now to state_prev

Once the graph machinery works, that could be automated.

How is this as a temporary solution?

[sbenthall]

I've added a few comments inline.

[Mv77]

• Tests for new functionality/models or Tests to reproduce the bug-fix in code.
• Updated documentation of features that add new functionality.
• Update CHANGELOG.md with major/minor changes.

[Mv77]

This is ready for re-review.

@sbenthall recommended me not to aim for the perfect solution that tried to go into aggregate states and their handling, but to go for whatever works now and revise once the higher-up design questions about states, aggregate, and post-states handling have been resolved. That was a great liberating piece of advise.

I did add a neat test. It creates a shock history, but then manually modifies its shocks and initial conditions to things that make it easy to check whether they are indeed being used. It then uses the shocks to simulate and compares the results with the easy-to-compute predicted behavior.

[llorracc]

@sbenthall, I guess you will review this again? Thx.

[sbenthall]

Awesome. Thanks for all this great work @Mv77 . I've reviewed and will merge the PR.