Add PyTable Paths to HDFStore #13

Closed
surbas opened this Issue Oct 7, 2010 · 4 comments

Comments

Projects
None yet
4 participants

surbas commented Oct 7, 2010

Hello,
I not sure if this was the best way to suggest improvements, but here I go anyway :)

I really like the easy of the HDFStore (and the entire project for that matter), but I wanted the ability to store and retrieve DataFrames in Groups below the root. .i.e

h5 = HDFStore('test.h5')
h5['/groups/below/theroot'] = DataFrame(data, index)

df =  h5['/groups/below/theroot']

I made the following changes to HDFStore class to do this, and am now using it in production code. If you agree this is something useful then I would like this to become apart of the main code base, using your own approach or the below.

I changed the repr, getitem, and _write_group functions in pandas.io.pytables.

def __repr__(self):
    output = str(self.__class__) + '\n'

    #Exstract path and kind of all 'pandas_type' pytable Groups.
    keys, values = zip(*((x._v_pathname, x._v_attrs.pandas_type) for x in self.handle.walkGroups() if hasattr(x._v_attrs,'pandas_type')))

    output += adjoin(5, keys, values)
    return output



def __getitem__(self, key):

    if not key[0] == '/': #Then add root slash so we can use getNode belwo
        key = '/' + key

    group = self.handle.getNode(key)
    return _read_group(group)



def _write_group(self, key, value):
    root = self.handle.root


    if key[0] == '/': #Assume they want a nested pytable Group
        final_slash = key.rfind('/')
        where = key[:final_slash]
        name = key[final_slash + 1:]
    else:
        where = '/'
        name = key

    try:
        group = self.handle.getNode(key)
    except:
        group = self.handle.createGroup(where, name, createparents=True)


    kind = type(value)
    handler = self._get_write_handler(kind)

    try:
        handler(group, value)
    except Exception:
        raise

    group._v_attrs.pandas_type = kind.__name__
    return True

please let me know if there is questions or concerns... I made it so that it still works just like the original if the user doesn't need groups...

Thanks for a great project.

Owner

wesm commented Jun 23, 2011

Hi Surbas,

I'm sorry this has taken me ages to get to. I just completed some major work on the HDFStore class. Let me see if I can incorporate your patch without breaking any of the unit tests, will get back to you.

Contributor

jreback commented Dec 9, 2012

@wesm I believe u can close this issue as 0.10 implements

Contributor

y-p commented Dec 12, 2012

@surbas, is there any functionality still missing in the latest git?
jreback has put in an enormous number of improvements into HDFS support
in 0.10.0.

surbas commented Dec 12, 2012

Looks good to me. Thanks for the work.

@surbas surbas closed this Dec 12, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment