hdf_archive key enumeration recursion error

[spotiris]

I have a HDF5 file I generated through hdf_archive which is a dictionary of dictionaries that include a few numpy arrays.
I would like to get the keys in the hdf_archive.
In dir_archive I can do:

list(archive.keys())

However the pickling process is quite slow and not scalable for the resources I have.
HDF5 provides fast random access, so I thought I would try hdf_archive, but I get the following error:

list(archive.keys())

RecursionError                            Traceback (most recent call last)
<ipython-input-114-7f77f4cb21ec> in <module>()
----> 1 list(archive_gt.keys())

/usr/lib/python3.5/_collections_abc.py in __iter__(self)
    651 
    652     def __iter__(self):
--> 653         yield from self._mapping
    654 
    655 KeysView.register(dict_keys)

... last 1 frames repeated, from the frame below ...

/usr/lib/python3.5/_collections_abc.py in __iter__(self)
    651 
    652     def __iter__(self):
--> 653         yield from self._mapping
    654 
    655 KeysView.register(dict_keys)

RecursionError: maximum recursion depth exceeded

[spotiris]

For those interested, a hack is to edit the hdf_archive class in _archives.py as below.
This doesn't return a KeysView object.

def keys(self):
          # if sys.version_info[0] >= 3:
          #     return KeysView(self) #XXX: show keys not dict
          filename = self.__state__['id']
          try:
              f = hdf.File(filename, 'r')
              if sys.version_info[0] >= 3:
                  _keys = [self._loadkey(bytes(key, 'ascii')) for key in f.keys()]
              else:
                  _keys = [self._loadkey(key) for key in self._attrs(f).keys()]
          except: #XXX: should only catch appropriate exceptions
              f = None
              raise OSError("error reading file archive %s" % filename)
          finally:
              if f is not None: f.close()
          return _keys

[mmckerns]

Sorry this has been ignored for so long... I can confirm this is an issue:

Python 3.7.16 (default, Dec  7 2022, 05:04:27) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import klepto as kl
>>> a = kl.archives.hdf_archive('foo.hdf', cached=True)
>>> a['a'] = 1
>>> a['b'] = 2
>>> a['c'] = [3,4]
>>> list(a.keys())
['a', 'b', 'c']
>>> a.dump()
>>> ^D

So, no issues when the archive serves as a backend to the cache.
However, if you use the archive directly...

Python 3.7.16 (default, Dec  7 2022, 05:04:27) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import klepto as kl
>>> a = kl.archives.hdf_archive('foo.hdf', cached=False)
>>> list(a.keys())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/_collections_abc.py", line 720, in __iter__
    yield from self._mapping
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/_collections_abc.py", line 720, in __iter__
    yield from self._mapping
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/_collections_abc.py", line 720, in __iter__
    yield from self._mapping
  [Previous line repeated 992 more times]
  File "/Users/mmckerns/lib/python3.7/site-packages/klepto/_archives.py", line 1885, in __iter__
    return iter(self.keys())
  File "/Users/mmckerns/lib/python3.7/site-packages/klepto/_archives.py", line 1866, in keys
    return KeysView(self) #XXX: show keys not dict
RecursionError: maximum recursion depth exceeded