accessing item from list within a list

[wdieter]

Not sure if an issue but was hoping to get some insight as to how you might specify this spec.
I have a dictionary that looks like this:

data = {"data": [{'name': 'bob',
                  'custom': {
                      'hobbies': [{
                        'swimming': True
                      }]
                    }
                  },
                 {'name': 'joey',
                  'custom': {
                      'hobbies': [{
                        'swimming': False
                      }]
                    }
                  }]
        }

and I want to get a dictionary that looks like:
{'names': ['bob', 'joey'], 'swims': [True, False]}
the closest I'm able to get is by using this spec:
spec = {"names": ("data", ["name"]), "swims": ("data", ["custom"], ["hobbies"], [['swimming']])}
but the 'swims' attribute comes back as [[True], [False]]
Is there a way to get those attributes out of those inner lists by adjusting the spec? It seems trivial but actually nested lists multiple levels deep seems pretty common in json.
BTW glom is super handy!
Thanks, W

[mahmoud]

Hey @wdieter! Great example. First off, I think you're absolutely right that unnesting and flattening are common operations that we're going to have to look at supporting natively. For this simple example, flattening lists can be achieved using the old sum() function:

glom(data, {"names": ("data", ["name"]), "swims": (("data", ["custom"], ["hobbies"], [['swimming']]), lambda t: sum(t, [])) })
# {'swims': [True, False], 'names': ['bob', 'joey']}

You'll notice all I added to your spec was that lambda at the end. That's one nice affordance of glom, you can always drop to a function. Still, if you run into more examples like these, please keep us posted, because they're invaluable when it comes to building the native functionality. Thanks again!

[kurtbrose]

T[0] can also help un-nest things if you know there is going to be exactly one item.

spec = {"names": ("data", ["name"]), "swims": ("data", ["custom"], ["hobbies"], [(['swimming'], T[0])])}
glom(data, spec)
# {'swims': [True, False], 'names': ['bob', 'joey']}

[uatach]

@wdieter sorry for the (almost) duplicate, I've missed your issue when creating mine.
@mahmoud @kurtbrose those helped a lot, I've managed to acomplish what I wanted.

[wdieter]

Thanks @mahmoud and @kurtbrose for the help there. I looked into the the issue a bit more and I'm not sure if the sum method works when there is more than one level of nesting or different levels of nesting, eg. [[['a']], [[['b']]]]. I didn't look into the solution utilizing T but my guess is you would also need to change it to something like [[(['swimming'], T[0][0])]]? This may work but the many layers of lists and tuples isn't so nice in my opinion. Also you already defined the access level in the first part of the spec so it feels a bit repetitive.

I think a flatten function should be able to handle one or more levels of nesting eg. both [['a'], ['b']] and [[['a']], [['b']]]. I found this recursive solution at https://stackoverflow.com/questions/2158395/flatten-an-irregular-list-of-lists that seems to work with one or many levels.

def flatten(x):
    if isinstance(x, collections.Iterable) and not isinstance(x, (str, bytes)):
        return [a for i in x for a in flatten(i)]
    else:
        return [x]

So if my dictionary now has three lists:

data2 = {"data":[{'name':'bob',
             'custom':{
                 'species':'person',
                 'hobbies': [{
                     'exercise': [{
                         "swimming": True}]
                     }]
                 }
            },
            {'name':'joey',
             'custom':{
                 'species':'cat',
                 'hobbies': [{
                     'exercise': [{
                         "swimming": True}]
                     }]
                 }
            }]
    }

My spec would be:

s2 = {"names": ("data", ["name"]),
"swims": ("data", ["custom"], ["hobbies"], [["exercise"]], [[['swimming']]],  lambda t: flatten(t))}

giving me

{'names': ['bob', 'joey'], 'swims': [True, False]}

And this flatten function works for the orginal dict too

[wdieter]

One more issue with sum(t, []) and also with the flatten func above: empty lists will be discarded. This can be an issue if you're trying to use glom to get a dict that can be passed to a dataframe and need to preserve consistent row counts where row[x] should represent one entity

In [23]: l = [['a'], [], ['b']]

In [24]: sum(l, [])
Out[24]: ['a', 'b']

My workaround:

In [25]: [item[0] if item else 'blank' for item in l]
Out[25]: ['a', 'blank', 'b']

[kurtbrose]

what about Coalesce(T[0])?

>>> glom([ [1], [], [3] ], [Coalesce(T[0], default=None)])
[1, None, 3]

[mahmoud]

I think we've got this one sorted, I'm going to close out the issue. Feel free to chat further, and a PR for a snippet to this effect would be great!