Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize Parameter inheritance #1605

Merged

Conversation

kmantel
Copy link
Collaborator

@kmantel kmantel commented May 6, 2020

This removes the need for most getattr calls using caching.

If a Parameter is marked as _inherited=True, then it does not store its default value (for example) explicitly but rather inherits the value in real time from its closest non-inherited parent. This was implemented by deletion of the attribute and repeated getattr calls to find the correct inherited value.

This branch implements storage of a reference to the closest uninherited parent, which is recalculated lazily in the uncommon event that the correct parent is changed (by explicitly setting the default value for a previously inherited parent, or by calling Parameter.reset() on an uninherited parent in some cases).

It also removes most getattr calls for Parameters, which now mostly only occurrs when setting up ParameterAliases

@jvesely
Copy link
Collaborator

jvesely commented May 6, 2020

sounds like this fixes #1500. @SamKG ?

@kmantel
Copy link
Collaborator Author

kmantel commented May 6, 2020

I'm not sure if it's completely minimized, but there aren't obvious places for improvement right now. For comparison on number of calls for just importing psyneulink:

devel:

ncalls tottime percall cumtime percall filename:lineno(function)
329679 0.2666 8.088e-07 0.4017 1.218e-06 parameters.py:865(_parent)
319674/85617 0.3139 3.667e-06 1.125 1.314e-05 parameters.py:795(getattr)
50765 0.05026 9.9e-07 0.188 3.703e-06 parameters.py:429(_owner)
49750/8153 0.3114 3.82e-05 0.6914 8.481e-05 parameters.py:1390(getattr)
33072 0.02012 6.083e-07 0.03099 9.371e-07 parameters.py:372(_is_parameter)
31878/19187 0.06516 3.396e-06 0.7309 3.809e-05 parameters.py:803(setattr)
8212 0.01563 1.903e-06 0.7334 8.93e-05 parameters.py:1467(_get_prefixed_method)
6888 0.04082 5.927e-06 0.7208 0.0001047 parameters.py:1503(_validate)
3948/3750 0.03547 9.459e-06 0.4676 0.0001247 parameters.py:1409(setattr)
3370 0.08262 2.452e-05 0.5885 0.0001746 parameters.py:842(_inherited)
3370 0.001303 3.866e-07 0.001303 3.866e-07 parameters.py:838(_inherited)

this branch:

ncalls tottime percall cumtime percall filename:lineno(function)
103828 0.08282 7.977e-07 0.1198 1.154e-06 parameters.py:931(_parent)
85617 0.1526 1.782e-06 0.2946 3.441e-06 parameters.py:808(getattr)
39727/24346 0.07239 2.974e-06 0.3629 1.49e-05 parameters.py:844(setattr)
33637 0.01936 5.756e-07 0.02994 8.902e-07 parameters.py:378(_is_parameter)
9908 0.009737 9.827e-07 0.02963 2.991e-06 parameters.py:435(_owner)
8869/8153 0.01675 2.055e-06 0.1042 1.278e-05 parameters.py:1456(getattr)
8432 0.0428 5.076e-06 0.08217 9.745e-06 parameters.py:1457(throw_error)
8212 0.01397 1.702e-06 0.1372 1.671e-05 parameters.py:1553(_get_prefixed_method)
6888 0.02694 3.912e-06 0.1663 2.414e-05 parameters.py:1589(_validate)
3948/3750 0.04085 1.089e-05 0.184 4.907e-05 parameters.py:1487(setattr)
3370 0.07794 2.313e-05 0.2991 8.875e-05 parameters.py:883(_inherited)
3370 0.001475 4.377e-07 0.001475 4.377e-07 parameters.py:879(_inherited)

@jvesely
Copy link
Collaborator

jvesely commented May 7, 2020

going by the "300000" calls in #1500. I'd say that the change 319674/85617 to 85617 addresses that, but I'd leave it to @SamKG to decide whether to close the issue.

@SamKG
Copy link
Collaborator

SamKG commented May 8, 2020

This is a pretty drastic improvement! Great work!

@jdcpni
Copy link
Collaborator

jdcpni commented May 8, 2020 via email

If a Parameter is marked as _inherited=True, then it does not store its default
value (for example) explicitly but rather inherits the value in real time
from its closest non-inherited parent. This was implemented by deletion of
the attribute and repeated getattr calls to find the correct inherited value.

This commit implements storage of a reference to the closest uninherited
parent, which is recalculated lazily in the uncommon event that the correct
parent is changed (by explicitly setting the default value for a previously
inherited parent, or by calling Parameter.reset() on an uninherited parent in
some cases).
Mainly to avoid nested getattr calls for parsing and validation methods. These
are not Parameter objects, and if they're present, they will be in the
Parameters.__dict__ and accessible normally.
Avoids keeping any deleted Parameter classes from garbage collection
@kmantel kmantel force-pushed the refac/parameters/optimize branch from 388379e to e4a34af Compare May 8, 2020 22:08
@kmantel kmantel merged commit 9067cd5 into PrincetonUniversity:devel May 8, 2020
@kmantel kmantel deleted the refac/parameters/optimize branch May 8, 2020 23:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants