DataFrame.dtypes

[AbdealiLoKo]

Raising this to discuss what the handling of the .dtypes should be.
Current implementations:

• Pandas: Returns a pandas Series with index of fieldnames and values as dtypes
• Spark: Returns a list of tuples with (fieldname, dtype)
• Koala: Same as Spark

Other Differences:

• Whether to show indexes in .dtypes:
  • Pandas: Shows only columns and not indexes
  • Spark: Shows all columns, no index concept
  • Koala: Shows columns and indexes
• Type values:
  • Pandas: Shows strings, arrays, map, etc as object
  • Spark: Shows strings, arrays, map, etc in appropriate simpleString notation
  • Koala: Same as spark
• Dtype notation:
  • Pandas: Shows numpy type for the dtype
  • Spark: Shows simpleString notation
  • Koala: Same as spark

[rxin]

It sort of depends on what users use dtypes for. If they are using it to programmatically get the column names and data types, then it'd make sense to be as much the same as Pandas as possible. If they are just using that to inspect the dataset's schema, then the existing one should be fine.

[AbdealiLoKo]

When you say inspect do you mean if they manually inspect it?
Wouldn't manually inspection be OK if it's pandas behaviour too?

[thunterdb]

Unless there are objections, I think that the behaviour for dtypes should be to return the same as pandas. Spark users will have schema if they want to have the underlying spark types.