Skip to content
This repository has been archived by the owner on Sep 1, 2022. It is now read-only.

Error when serializing scala spark DataFrame that comes out of a sqlContext.read.* #72

Closed
lbustelo opened this issue Oct 29, 2015 · 1 comment

Comments

@lbustelo
Copy link
Collaborator

See https://gist.github.com/lbustelo/42f074827738ba45c0de for example.

It seems that when creating DataFrames using the sqlContext.read.* method we get an error with serialization.

/cc @vinomaster @wellecks

@lbustelo
Copy link
Collaborator Author

The problem has to do with our serialization of Dataframes. We are running
``df.take(limit).map(row => row.toSeq.toArray.map(_.toString))`

But if the value is null (not present) we get an NPE.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant