New node cache (compatibility with Python 3.4) #311
Conversation
Now they behave a little bit more like a dict.
|
This looks great to me, but I haven't tested it yet. I'll try to get to that tonight. Thanks @avalentino! |
|
Great job, thanks for this @avalentino. |
| class _DictCache(dict): | ||
| def __init__(self, nslots): | ||
| if nslots < 1: | ||
| raise ValueError("Invaòlid number of slots: %d" % nslots) |
|
Excellent job Antonio. Besides some syntactic errors, the only thing that I noticed is that a negative value for NODE_CACHE_SLOTS is not supported anymore. Have you added some check in code for detecting that? Some users may be using that. |
| However, and in order to avoid a large memory consumption, the user will be | ||
| warned when the number of loaded nodes will reach the -NODE_CACHE_SLOTS | ||
| warned when the number of loaded nodes will reach the ``-NODE_CACHE_SLOTS`` | ||
| value. |
There was a problem hiding this comment.
Uh, is this inconsistent with https://github.com/PyTables/PyTables/pull/311/files#diff-0035a6b5c4e6a6c60fe4778f61bca5b2L170 where you explicitly removed the docs about negative values of NODE_CACHE_SLOTS or am I missing something?
The possibility to use a DictCache has been re-introduced in 310b355.
|
Hi @FrancescAlted, thank you very much for the code review
Ah sorry, I re-introduced that feature in 310b355 but forgot to revert the change in the doc. |
|
#121 mentions NODE_CACHE_SLOTS too, can that be addressed here too? |
yes, it would be nice. |
|
Hi @andreabedini, just closed #121. |
This PR implements a possible solution to the incompatibility (see #306) of the node cache management currently implemented in Pytables 3.0 with the new object finalization strategy (see [1], [2]) introduced in Python 3.4 (tested on Python 3.4b1).
The node caching mechanism has been completely redesigned to be simpler and less dependent from specific behaviors of the
__del__method.A new NodeManager class has been introduced that takes in charge all stuffs related to node caching and node finalization.
It includes a LRU cache for a fast node access and a
registrythat holds a (weak) reference to all open nodes (including the ones in the cache) to ensure a correct finalization when the file handler is closed.The new implementation seems to work with all supported python versions (including Python 3.4 tip).
but no performance test has been performed (feedback is welcome).
P.S. probably the documentation regarding
NODE_CACHE_SLOTSshould be updated to reflect the new implementation.[1] http://docs.python.org/3.4/whatsnew/3.4.html#pep-442-safe-object-finalization
[2] http://www.python.org/dev/peps/pep-0442