the "whiteboard problem"

[sbenthall]

Throughout HARK, the AgentType object is being used like a whiteboard.

Mathematical objects are assigned as attributes to self willy-nilly. This leads to an unstructured namespace that is hard to understand.

As a further refactoring step, aligned with #660 and other issues:

• compartmentalizing the math so that more of the board is "erased" between operations
• when possible, the math "inputs" into functions get passed in as arguments rather than getting drawn from self.

This will encapsulate the code in a way that makes it easier to decompose and refactor.

[sbenthall]

Another way to address this is to introduce more namespaces.

A good target for this might be the many *Dstn variables that get created. I.e. IncomeDstn, RiskyDstn.
These then get sampled for shocks.

This connects to the idea that maybe these distributions should be passed in directly as parameters: #620

But other namespaces could be introduced for, for example, mathematical objects used for checkConditions, or those used when computing solutions.

[sbenthall]

I had an idea that's a little goofy, but which may be a good short term solution.

This is to add a property, "boards", to the AgentType object.
Then a new "whiteboard" can be made for the mathematical work of a particular method by doing this:

self.boards["myContext"] = (board := {})

It can also be written without the Python 3.8 walrus operator but you get the idea.

Then in that method one can assign properties to the board:

board.MPCnow = #whatever

If you really want to inspect the mathematical work, then you can find the whiteboard where it was written and look at it. But this way it won't clutter the top-level namespace.

I would much prefer a functional programming orientation and I think we need to get there for complete mathematization of Frames. But I think this would be an improving to the current HARK model code that would help get us to the Frame architecture by structuring the data little more.

[sbenthall]

What about a decorator (@etc) for importing a namespace into the scope of a method so that the mathematical objects are available for use?

[sbenthall]

Punting to 1.1 for time reasons.

[sbenthall]

Considering #1410 ---

When a model is initialized, it is passed a parameters dictionary.

The values of this dictionary used to be copied over onto the object as attributes (i.e. using setattr), and many other computed mathematical values were stored in this way.

This is bad for a number of reasons. For example, it makes it very hard to find all the parameters, since they exist alongside all the other attributes of an object.

So some changes were made: The model was changed to store the parameters dictionary as an argument. @alanlujan91 then made a dedicated parameters object.

But because there was so much code that depended on referencing arbitrary things on the model object, I didn't bother to go through and change all the cases where parameters were referenced in this way. That's why there's currently both, redundant, ways of storing parameters on a model.

To be clear, this means that on a model object, parameters live in both the self.parameters dictionary, and the object itself. Also on the object itself are things that have been constructed mathematically.

Never before have things that have been constructed mathemaically been added to the parameters dictionary, because that dictionary was just a copy of the values the model was initialized with.

Often it's been floated that constructed objects should live in a dedicated namespace. I believe @llorracc used bilt for something like this in his pre-alpha implementation.

I don't know why @mnwhite would like to add 'constructs' to the parameters dictionary in #1410 ; it seems like a bad idea to me. But he seems to have his reasons, which are perhaps in the short term.

In the longer term, I don't think we should have any of these values on the object. Rather, it would be better to have namespaces. Understanding which namespace we actually need would go a long way towards rationalizing our model designs.

[llorracc]

I’m strongly in favor of namespaces to ameliorate the whitespace problem As Seb recalls, in my pre-alpha I proposed a bilt namespace to segregate objects that were constructed from those that were exogenous. I had also distinct namespaces for exogenous parameters, and I think for expectational objects that were constructed. Like, nexpectations were expectations of objects from the next period constructed by expectoration. I believe I also had “prospectations” for beginning-of-period objects. Anyway, I’m in favor of thinking carefully about what objects belong in what namespaces.

…

On Wed, May 15, 2024 at 6:34 PM Sebastian Benthall ***@***.***> wrote: Considering #1410 <#1410> --- When a model is initialized, it is passed a parameters dictionary. The values of this dictionary used to be copied over onto the object as attributes (i.e. using setattr), and many other computed mathematical values were stored in this way. This is bad for a number of reasons. For example, it makes it very hard to find all the parameters, since they exist alongside all the other attributes of an object. So some changes were made: The model was changed to store the parameters dictionary as an argument. @alanlujan91 <https://github.com/alanlujan91> then made a dedicated parameters object. But because there was so much code that depended on referencing arbitrary things on the model object, I didn't bother to go through and change all the cases where parameters were referenced in this way. That's why there's currently both, redundant, ways of storing parameters on a model. To be clear, this means that on a model object, parameters live in *both* the self.parameters dictionary, and the object itself. Also on the object itself are things that have been constructed mathematically. Never before have things that have been constructed mathemaically been *added* to the parameters dictionary, because that dictionary was just a copy of the values the model was initialized with. Often it's been floated that constructed objects should live in a dedicated namespace. I believe @llorracc <https://github.com/llorracc> used bilt for something like this in his pre-alpha implementation. I don't know why @mnwhite <https://github.com/mnwhite> would like to add 'constructs' to the parameters dictionary in #1410 <#1410> ; it seems like a bad idea to me. But he seems to have his reasons, which are perhaps in the short term. In the longer term, I don't think we should have any of these values on the object. Rather, it would be better to have namespaces. Understanding which namespace we actually need would go a long way towards rationalizing our model designs. — Reply to this email directly, view it on GitHub <#889 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKCK7YZE4XOGMDEZEGIRSDZCPPIVAVCNFSM4VERM3MKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMJRGM2TSMZVGEZA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- - Chris Carroll

[mnwhite]

It seems I misunderstood what the purpose of the parameters dictionary was. I thought it was meant to be the replacement for where all of the data/information about the instance's problem was supposed to go, rather than being in the top level of the object. That is, anything that would be used by its solver or simulator, as well as anything that will be used to make something that will be used by the solver or simulator (recursively, ad infinitum), should be in the parameters dictionary.

That seemed to be the intent of Mateo's PR in #1389 , but maybe I misunderstood that as well.

The reason I put constructed objects in the same place(s) as "raw" parameters is that there's no distinction between them, and no way to tell them apart. As an example, the default format of PermGroFac for IndShockConsumerType (and all of our models) is that you pass it a list (or array) of floats, one for each period; it defaults to a expecting raw input ready for use in the various solve_one_period functions. However, in #1410 , a user could do the following:

MyType.constructors['PermGroFac'] = make_polynomial_PermGroFac
MyType.T_cycle = 40
MyType.PermGroFac_coeffs = [1.002, 0.001, -0.00003]
MyType.construct('PermGroFac')

(All of that could be done in the dictionary passed at initialization as well.) That will construct a 40 period list for PermGroFac using polynomial coefficients (here, quadratic) on t. Where should that list be stored? In a dictionary like bilt or constructs? Or in parameters, where it would go if we had passed a hand-written PermGroFac list... or one that was made by calling make_polynomial_PermGroFac outside of the agent using those same inputs! That's what I mean by there being no difference between "raw" parameters and constructed ones.

Likewise, I would like for the solver and simulator to look in only one place for the information it needs, rather than checking several dictionaries (etc). Overall, I agree that we should think carefully about the organization of information and data structures, but the taxonomy of objects, and what they'll be used for, isn't always clean.

I have some ideas for how to make parameter information more accessible.

[llorracc]

On Wed, May 15, 2024 at 9:43 PM Matthew N. White ***@***.***> wrote: It seems I misunderstood what the purpose of the parameters dictionary was. I *thought* it was meant to be the replacement for where all of the data/information about the instance's problem was supposed to go, rather than being in the top level of the object. That is, anything that would be used by its solver or simulator, as well as anything that will be used to make something that will be used by the solver or simulator (recursively, ad infinitum), should be in the parameters dictionary. That seemed to be the intent of Mateo's PR in #1389 <#1389> , but maybe I misunderstood that as well. The reason I put constructed objects in the same place(s) as "raw" parameters is that there's no distinction between them, and no way to tell them apart.

The "whiteboard" problem is that there are too damn many things jumbled together with no organization or order. Breaking the contents up into subcomponents that are all of one conceptual type (not "python variable type") is valuable in helping navigate. It was hard for ME when writing the pre-alpha to find the variables I needed, until I split things up into distinct namespaces.

As an example, the default format of PermGroFac for IndShockConsumerType (and all of our models) is that you pass it a list (or array) of floats, one for each period; it defaults to a expecting raw input ready for use in the various solve_one_period functions. In #1410 <#1410> , a user could do the following: MyType.constructors['PermGroFac'] = make_polynomial_PermGroFac MyType.T_cycle = 40 MyType.PermGroFac_coeffs = [1.002, 0.001, -0.00001] MyType.construct('PermGroFac') (All of that could be done in the dictionary passed at initialization as well.) That will construct a 40 period list for PermGroFac using polynomial coefficients (here, quadratic) on t. Where should that list be stored?

NOT in the namespaces for, say, period 23. (and 24 and 22). In a dictionary like bilt or constructs?

Don't call them dictionaries. They're namespaces, and they are specific to the solution[23] object or whatever.

Or in parameters,

You mean, solution[23].parameters, right? Remember my principle that EVERY PERIOD IS CONSTRUCTED. Probably solution[23].parameters will be mostly identical with solution[24].parameters except for a few things like (in a life cycle model) that solution[23].parameters will have a different value of PermGroFac than solution[24].parameters.PermGroFac because the agents are a different age. where it would go if we had passed a hand-written PermGroFac list... or one

that was made by calling make_polynomial_PermGroFac *outside* of the agent using those same inputs! That's what I mean by there being no difference between "raw" parameters and constructed ones.

I think we need to have a long conversation on this subject because apparently our understandings of the whole topic have diverged. Likewise, I would like for the solver and simulator to look in only one

place for the information it needs, rather than checking several dictionaries (etc).

Namespaces. Why? It's trivial to combine namespaces. And if you really want to recreate the whole whiteboard you can create a namespace that is the merger of all the specific namespaces. Maybe we need to take another look at my pre-alpha.

…

Overall, I agree that we should think carefully about the organization of information and data structures, but the *taxonomy* of objects, and what they'll be used for, isn't always clean. I have some ideas for how to make parameter information more accessible. — Reply to this email directly, view it on GitHub <#889 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKCK7Z44W2EL4P2FI7J3A3ZCQFKRAVCNFSM4VERM3MKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMJRGM3TMNJVHE3A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- - Chris Carroll

[mnwhite]

I'm not sure we're using "namespace" in the same way. To me, a "namespace" in Python is more or less synonymous with a scope: it's a level at which object references are defined. A namespace or scope is created every time you call a function; objects and references created inside that namespace have access to references in the namespace in which it was created (recursively), and that namespace vanishes from existence when the function ends. Only the returned objects continue to exist (as well as anything in the function scope that ended up referenced by something outside the scope, but that's a somewhat unusual thing to do).

I think we might be talking about two separate things entirely. I'm only addressing how to store and structure things in the current version of HARK, to make incremental progress at improving it and preparing it for big changes in the semi-near future. I think you're talking about a much different future version of HARK.