Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

writing and reading dataframes with lists as members #2158

Closed
zachcp opened this Issue · 4 comments

5 participants

zachcp Wes McKinney Andy Hayden pondycrane jreback
zachcp

I am not sure if this is a feature or a bug but when I use the default to_csv() and read_csv() functions on dataframes that have lists as members, I lose the lists.

This is a copy of this stack overflow post.

df1 = DataFrame({'a':[['john quincy', 'tom jones', 'jerry rice'],['bob smith','sally ride','little wayne'],['seven','eight','nine'],['ten','eleven','twelve']],'b':[9,2,4,5], 'c': [7,3,0,9]})
df1.to_csv('temp.csv')
df2 = read_csv('temp.csv')

#note the list(df1) is now a string(df2)
df1['a'][0]
['john quincy', 'tom jones', 'jerry rice']
df2['a'][0]
"['john quincy', 'tom jones', 'jerry rice']"

as an answer in that post showed, the list can be recovered using ast:

import ast
df2['a']=df2['a'].apply(lambda x: ast.literal_eval(x))
df2['a'][1][1]
'sally ride'

Can the pandas read/write parsers do this automatically without needing this ast step?

Wes McKinney
Owner

My best guess is "probably not"-- it's a difficult type inference problem. I'll leave this issue open for a while, though

Andy Hayden
Collaborator

It's been a while :)

As you say, the standard trick here is to apply ast.literal_eval. Don't think you can infer this as it could actually be a string...

Andy Hayden hayd closed this
pondycrane

I've been stucked with this problem as well

jreback
Owner

@pondycrane using a non scalar as a Dataframe element is almost not the right way to represent data
it's just not performant in any way and the api is horrible

try using MultiIndexs or Panel if u need more dimensions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.