Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-13098][SQL] remove GenericInternalRowWithSchema #10992

Closed
wants to merge 1 commit into from

Conversation

cloud-fan
Copy link
Contributor

This class is only used for serialization of Python DataFrame. However, we don't require internal row there, so GenericRowWithSchema can also do the job.

@SparkQA
Copy link

SparkQA commented Jan 30, 2016

Test build #50418 has finished for PR 10992 at commit 57707df.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor Author

retest this please

@SparkQA
Copy link

SparkQA commented Jan 30, 2016

Test build #50423 has finished for PR 10992 at commit 57707df.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor Author

retest this please

@SparkQA
Copy link

SparkQA commented Jan 30, 2016

Test build #50428 has finished for PR 10992 at commit 57707df.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor Author

retest this please.

@SparkQA
Copy link

SparkQA commented Jan 30, 2016

Test build #50434 has finished for PR 10992 at commit 57707df.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor Author

cc @davies

@davies
Copy link
Contributor

davies commented Jan 30, 2016

The difference between Row and InternalRow is that the objects inside InternalRow should be internal types. For this case, we pickle the objects in internal types, so it should be InternalRow (it's more clear even with some duplicated codes).

@cloud-fan
Copy link
Contributor Author

in toJava, we convert internal types to exteral ones(e.g. UTF8String -> String, ArrayData -> ArrayList), the InternalRow is the only exception, we convert it to GenericInternalRowWithSchema! I think using GenericRowWithSchema is more clear for it.

@davies
Copy link
Contributor

davies commented Jan 30, 2016

@cloud-fan I see, that make sense. LGTM, merging this into master, thanks!

@davies
Copy link
Contributor

davies commented Jan 30, 2016

@cloud-fan btw, it's not external type either, for example, it's Int for DateType, not java.sql.Date.

@asfgit asfgit closed this in dab246f Jan 30, 2016
@cloud-fan cloud-fan deleted the python branch January 30, 2016 08:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants