refactor getStates to use a transition function

[sbenthall]

One of (perhaps the only?) representation of the transition function in HARK models is in the getStates() method, which is called in the main simulation loop.

For example:

HARK/HARK/ConsumptionSaving/ConsRepAgentModel.py

Line 221 in 455d09e

def getStates(self):

It would simply be a matter of refactoring to extract the mathematics within this method into a single function of the same type signature as Dolo's transition function: https://dolo.readthedocs.io/en/latest/model_specification.html#transitions

If done in this way, it would be easier to do the following:

• start with a Dolang/YAML model document
• parse the document into a Dolo model, which would include a transition function object
• pass that transition function object into HARK

That would make it so the standard YAML representation of the model was not merely superficially connected to the HARK object, but also in an internal way.

[mnwhite]

`getPostStates` is also a candidate for this kind of conversion, as it's also a purely mechanical step based on the model definitions: `getStates` takes post-states (from t-1) and shocks to get today's states `getPostStates` takes states and controls to get today's post-states The methods `getShocks`, `getControls`, `simDeath` and `simBirth` are also decent candidates, but are a bit more complicated because they either (a) involve distribution objects or (b) have varying representations of the policy functions.

…

On Thu, Jul 16, 2020 at 1:33 PM Sebastian Benthall ***@***.***> wrote: One of (perhaps the only?) representation of the transition function in HARK models is in the getStates() method, which is called in the main simulation loop. For example: https://github.com/econ-ark/HARK/blob/455d09e44306e3bc24edf028c0daad3f6968b364/HARK/ConsumptionSaving/ConsRepAgentModel.py#L221 It would simply be a matter of refactoring to extract the mathematics within this method into a single function of the same type signature as Dolo's transition function: https://dolo.readthedocs.io/en/latest/model_specification.html#transitions If done in this way, it would be easier to do the following: - start with a Dolang/YAML model document - parse the document into a Dolo model, which would include a transition function object - pass that transition function object into HARK That would make it so the standard YAML representation of the model was not merely superficially connected to the HARK object, but also in an internal way. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#761>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADKRAFPJ7QZHI653U54AKDTR342XPANCNFSM4O4UMJXQ> .

[sbenthall]

I believe 'getShocks' corresponds to what Dolo splits off into a separate collection, exogenous. I guess because it does not require computing state transitions, it can be more efficiently implemented as a standalone process?

The shocks values are then an input in the transition function, in the Dolo spec.

This relates to #760. Sorry I missed your note about bringing it up on the call.

[sbenthall]

I don't think Dolo does death and birth. I believe this is a substantive difference between HARK and Dolo at this point.

[albop]

Indeed, in dolo, implicitely we live and let die. Currently the problem you can represent in dolo is a unique stationary process. There is currently nothing in it to represent a non-stationary process or the evolution of a population. What you have in "transition" is the detrended/conditional version of the model.

…

On Thu, Jul 16, 2020, 7:43 PM Sebastian Benthall ***@***.***> wrote: I don't think Dolo does death and birth. I believe this is a substantive difference between HARK and Dolo at this point. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#761 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACDSKPB4NPAJTYKUAQT36DR3434TANCNFSM4O4UMJXQ> .

[sbenthall]

I'm working on this issue.

One thing I'm running into: it looks like:

• some state variables are stored with a separate value for each agent in a simulation,
• and some are meant to be scalars

but there's currently no way this difference is tracked structurally in the code. It is handled by the way individual variables are coded.

Example: it looks like pLvlNow has a value for each agent, but PlvlAggNow is a scalar? I guess because it is an aggregate value?

I'm working with the PerfectForesight model, which is supposed to be a quite basic case. But it looks like the aggregate state variable tracking here is breaking the MDP abstraction.

I'm going to need to write some generalized support for this, I suppose. I wonder what thoughts others have, especially @mnwhite .

[llorracc]

Example: it looks like pLvlNow has a value for each agent, but PlvlAggNow is

a scalar? I guess because it is an aggregate value? Exactly: Everybody experiences the same aggregate state. Having it be a scalar prevents potential bugs were different people were living in different aggregate states at the same time.

On Mon, Sep 14, 2020 at 12:08 PM Sebastian Benthall < ***@***.***> wrote: I'm working on this issue. One thing I'm running into: it looks like: - *some* state variables are stored with a separate value for each agent in a simulation, - and *some* are meant to be scalars but there's currently no way this difference is tracked structurally in the code. It is handled by the way individual variables are coded. Example: it looks like pLvlNow has a value for each agent, but PlvlAggNow is a scalar? I guess because it is an aggregate value? I'm working with the PerfectForesight model, which is supposed to be a quite basic case. But it looks like the aggregate state variable tracking here is breaking the MDP abstraction.

"breaking the MDP abstraction.' Not sure what you mean by that. There's nothing about being a scalar that prevents the aggregate state from being a state...

…

I'm going to need to write some generalized support for this, I suppose. I wonder what thoughts others have, especially @mnwhite <https://github.com/mnwhite> . — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#761 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKCK75TON55ST3SIUGSPXTSFY5YHANCNFSM4O4UMJXQ> .

-- - Chris Carroll

[sbenthall]

"breaking the MDP abstraction.'

I was under the impression that an agent simulated within an AgentType was an isolated run of an MDP. The MDP formalism has no information exchanged between different runs.

Now it sounds like you are saying that it is typical for the simulation of multiple agents within an AgentType to share information over the course of their simulated run.

Technically, I believe that makes it a Multi-Agent Markov Decision process (MMDP) of some kind, because each agent is making decisions based on an individual state, and there's also aggregated state. I thought this extra level of complexity was introduced only in the Market class.

I wonder how dolo deals with aggregated state like this. @albop ?

[albop]

I"m honestly not sure I follow the discussion. Here is the model in dolark:

• to solve for each individual's decision rule, we have one individual-specific approximated exogenous process (the exogenous states today, and where they can be tomorrow with appropriate transition probabilities). This process is perceived by the individual and can incorporate both an idiosyncratic and an aggregate component
  • solving for this problem yields the solution for each agent as a regular MDP problem given the exogenous shock so this solution part is conceptually independent across all agents but...
  • the aggregate component of the exogenous process can depend on every agent's decision agents so...
  • one needs to solve at the same time for all agents, or iterate somehow between the agregate process and the agent's behaviour. This makes the whole problem some instance of a multi-agent class of problem though I wouldn't call it MMDP, since there are no direct interactions between agents. The closest denomination I know is Mean Field Game, which might not be completely accurate either.
• to simulate the population of all agents, there are two cases:
  • either the aggregate variables are time invariant, in which case one can simulate all agents separately, or do something smarter
  • or the aggregate variables fluctuate over time, and depend on the agents' positions. Everything needs to be simulated at once.

I'm still not sure this casuistic distinctions contribute to the ongoing discussion. Can you precise the question a bit more?

[llorracc]

Now it sounds like you are saying that it is typical for the simulation

of multiple agents within an AgentType to share information over the course of their simulated run. Technically, I believe that makes it a Multi-Agent Markov Decision process (MMDP) of some kind, because each agent is making decisions based on an individual state, and there's also aggregated state. I thought this extra level of complexity was introduced only in the Market class. Well, yes, versions where you don't need to use the Market class are versions where there is no meaningful equilibrium between the agents. But for research purposes, almost all uses of the toolkit are likely to need to be aggregate/macro models (which is to say, they will use the Market class). Many of our DemARKs and most examples do not use the whole Market class because they are designed to be focused explorations of specific points, and if the point in question is not meaningfully affected by general equilibrium (="Market") considerations, there's no point including the extra complexity of building them in a GE framework. But I'm guessing that your question is motivated somehow by some practical concern about implementation, but I think neither Pablo nor I has a clear sense of what that is. "to share information over the course of their simulated run" ... Well, kind of. I'd say that the chief distinction between what economists typically do in these kinds of simulations, and what "ABM" people do, is that economists assume that people interact ONLY with the market, whose equilibrium is the result of everybody's collective actions. No single individual has any meaningful effect on any other individual. So, if you were worried about needing to build some kind of direct-interaction framework - no. (Though it is an ambition that eventually it would be nice to have simulation tools that allow for this).

…

On Mon, Sep 14, 2020 at 4:14 PM Sebastian Benthall ***@***.***> wrote: "breaking the MDP abstraction.' I was under the impression that an agent simulated within an AgentType was an isolated run of an MDP. The MDP formalism has no information exchanged between different runs. Now it sounds like you are saying that it is typical for the simulation of multiple agents within an AgentType to share information over the course of their simulated run. Technically, I believe that makes it a Multi-Agent Markov Decision process (MMDP) of some kind, because each agent is making decisions based on an individual state, and there's also aggregated state. I thought this extra level of complexity was introduced only in the Market class. I wonder how dolo deals with aggregated state like this. @albop <https://github.com/albop> ? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#761 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKCK76M26JROAT44KG2PVDSFZ2QZANCNFSM4O4UMJXQ> .

-- - Chris Carroll

[sbenthall]

Thank you. This is helpful.

It is certainly helpful for me if I understand the underlying mathematics of the models.
In refactoring HARK towards 1.0, I'm hoping we can get a cleaner correspondence between the software architecture and the generalizable mathematics.

So it's useful to know that these are not-exactly-MDPs.

I see what you mean about it being different if the aggregate variables are exogenous or endogenous.

Maybe there's different formal model classes going on:
(1) MDP
(2) MDP with aggregate exogenous state
(3) MDP with aggregate endogenous state

My impression was that type (3) simulations in dolo required the dolark extensions to allow for the definition of a projection function, as in here:
https://github.com/EconForge/dolark/blob/master/examples/ks.yaml

Is there an example of a dolo simulation that does a type (2) model? How do you specify the aggregated exogenous state in the YAML?

[sbenthall]

is that economists assume that people interact ONLY with the market, whose equilibrium is the result of everybody's collective actions

Hmmm. The use of PlvlAggNow in the PerfectForesightModel appears to be an exogenous process and, in that particular case, not a stochastic one.

I can see why given that it's not stochastic, and because it was model that was written early on, it made sense to make the 'shortcut' of writing it in as a state variable of the agent.

But thinking about the cleanest future architecture, it seems like an important point whether this is exogenous state that is shared across all agents in a simulation, or exogenous state that can vary per agent.

I have a small hack around this, so it's not blocking me in the short run.
But I think it would be better if, when HARK users were specifying a model, there were the cues /documentation/architecture in place so they could make an explicit choice about this. (As opposed to the implicit one that's currently being based on the specific model code).

[albop]

(2) MDP with aggregate exogenous state

The aggregate state is always endogenous for interesting models. But it can be time-invariant, like the constant interest rate of the Ayiagari model. This one is also projected as a constant parameter (to be detemined) into each agents' problem (cf: https://github.com/EconForge/dolark/blob/master/examples/ayiagari.yaml)

[sbenthall]

The aggregate state is always endogenous for interesting models.

There appear to be some models in HARK that are of type (2).

It sounds like Dolo does not support this.

I will leave it to the economists to decide whether or not they are interesting in their terms.

[llorracc]

You are not getting the distinction between "the aggregate state" -- which is the configuration of ALL the aggregate state variables -- and "an aggregate state variable" -- which is the value of a single state variable. Like, ALL the aggregate states would include some measure of the distribution of wealth -- which is endogenous to all the past shocks. It would ALSO include what could be an exogenous state "boom" or "bust" which is determined by (exogenous) solar radiation ("sunspots"). "The aggregate state" encompasses the collection of all of them, exogenous and endogenous. And any collection in which ANY variables are endogenous is an endogenous collection (an "endogenous aggregate state").

…

On Mon, Sep 14, 2020 at 5:32 PM Sebastian Benthall ***@***.***> wrote: The aggregate state is always endogenous for interesting models. There appear to be some models in HARK that are of type (2). It sounds like Dolo does not support this. I will leave it to the economists to decide whether or not they are interesting in their terms. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#761 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKCK76JXRTMGJ7NSWFRLV3SF2DXXANCNFSM4O4UMJXQ> .

-- - Chris Carroll

[sbenthall]

Ah, ok. Thank you for that clarification.

As I understand it:

• For an MDP (type 1), there are endogenous and exogenous variables.
• For an Type 2 (whatever it is), there are endogenous and exogenous variables, and at least one aggregate exogenous variable.
• For Type 3, there are endogenous and exogenous variables, as well as at least one aggregate endogenous variable (and possibly aggregate exogenous variables as well).

What surprised me about PlvlAggNow--surprise that appears to be of surprise to you as well, given that it is such an edge case--is that it was it was an example of an aggregate exogenous variable without any other endogenous aggregate variables.

Hence, it was aggregation without a Market class involved.

I am confused now whether aggregate variables always come from the market or if they may also come from, e.g., solar radiation.

I also want to clarify that I am only including in this variables that have some downstream effect on the endogenous variables or controls. There are some cases where HARK computes and tracks a variable for monitoring or analyis purposes. I would call these "epiphenomenal variables" myself, due to some perhaps esoteric training in other fields.

[llorracc]

• For an MDP (type 1), there are endogenous and exogenous variables.

I think you meant to include "idiosyncratic" here?

• For an Type 2 (whatever it is), there are endogenous and exogenous variables, and at least one aggregate exogenous variable.

This is what we tend to call a "partial equilibrium" macro model.

• For Type 3, there are endogenous and exogenous variables, as well as at least one aggregate endogenous variable (and possibly aggregate exogenous variables as well).

Again, I think your first phrase meant to be "there are endogenous and exogenous idiosyncratic variables"

as well as at least one aggregate endogenous variable

e.g. aggregate wealth

(and possibly aggregate exogenous variables as well).

e.g., whether the economy is in a "boom" or "bust" state (or, the value of some aggregate productivity shock)

What surprised me about PlvlAggNow--surprise that appears to be of surprise to you as well, given that it is such an edge case--is that it was it was an example of an aggregate exogenous variable without any other endogenous aggregate variables.

It's not an edge case -- it's a very common setup. "Partial equilibrium" models are increasingly popular, because the profession has finally realized that

1. Often the results are almost the same in partial and general equilibrium
2. Doing full HA general equilibrium is massively more work.

Hence, it was aggregation without a Market class involved.

Right -- that's what "partial equilibrium" basically means -- you haven't imposed a Market equilibrium.

I am confused now whether aggregate variables always come from the market or if they may also come from, e.g., solar radiation.

There's not any sense in which exogenous aggregate variables need to "come from" the Market class -- as illustrated by our various partial equilibrium models with aggregate shocks. If you are solving a GE model, it might be tidy to bundle together the treatment all of the aggregate variables, exogenous and endogenous, in the part of the code that deals with the Market mechanism, but there is no necessity for doing so with respect to exogenous aggregate variables.

I also want to clarify that I am only including in this variables that have some downstream effect on the endogenous variables or controls. There are some cases where HARK computes and tracks a variable for monitoring or analyis purposes. I would call these "epiphenomenal variables" myself, due to some perhaps esoteric training in other fields.

Except possibly for debugging/diagnostic purposes, it's hard to see why "epiphenominal" variables would ever be interesting. Your definition of them basically is that these are variables that have no economic consequence.

[sbenthall]

I think you meant to include "idiosyncratic" here?

I think I see your point!
For a single MDP, "idiosyncratic" adds no new information.

But when it's distinguishing from aggregates, I see: yes, there should be an "idiosyncratic" there.

It's not an edge case -- it's a very common setup. "Partial equilibrium" models are increasingly popular

Ah, thank you. This is hugely clarifying.

Except possibly for debugging/diagnostic purposes, it's hard to see why "epiphenominal" variables would ever be interesting.

And yet, they are sometimes being tracked in the HARK code.
Presumably for debugging or diagnostic purposes.
This is why I brought it up.

In this and the other active thread, I'm trying to get at terminology that can help clarify what's going on in the software. It may make it into documentation or more scholarly writeups.

But maybe there are interesting cases for this. Maybe there's a model of an economy where, say, inequality measured by the Gini coefficient has to endogenous effect, but is nevertheless of policy of research interest.

[llorracc]

But maybe there are interesting cases for this. Maybe there's a model of

an economy where, say, inequality measured by the Gini coefficient has to endogenous effect, but is nevertheless of policy of research interest. Good example. The distribution of wealth itself matters, but a single summary statistic of it does not have any independent effect.

…

On Wed, Sep 16, 2020 at 2:48 PM Sebastian Benthall ***@***.***> wrote: I think you meant to include "idiosyncratic" here? I think I see your point! For a single MDP, "idiosyncratic" adds no new information. But when it's distinguishing from aggregates, I see: yes, there should be an "idiosyncratic" there. It's not an edge case -- it's a very common setup. "Partial equilibrium" models are increasingly popular Ah, thank you. This is hugely clarifying. Except possibly for debugging/diagnostic purposes, it's hard to see why "epiphenominal" variables would ever be interesting. And yet, they are sometimes being tracked in the HARK code. Presumably for debugging or diagnostic purposes. This is why I brought it up. In this and the other active thread, I'm trying to get at terminology that can help clarify what's going on in the software. It may make it into documentation or more scholarly writeups. But maybe there are interesting cases for this. Maybe there's a model of an economy where, say, inequality measured by the Gini coefficient has to endogenous effect, but is nevertheless of policy of research interest. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#761 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKCK7236WZOVW6L4QLJOGLSGEB7PANCNFSM4O4UMJXQ> .

-- - Chris Carroll

[sbenthall]

Fixed with #836