Skip to content

Liang-Chi Hsieh
viirya

Organizations

@SparkTC
Jun 27, 2016
viirya commented on pull request apache/spark#13778
@viirya

ok. let me try it.

Jun 27, 2016
viirya commented on pull request apache/spark#13778
@viirya

No. The test will be failed.

Jun 27, 2016
viirya commented on pull request apache/spark#13778
@viirya

ok. let me check it.

Jun 27, 2016
Jun 27, 2016
viirya commented on pull request apache/spark#13778
@viirya

when will the inputData.dataType be python udt? The first bug I showed in the PR description.

Jun 27, 2016
viirya commented on pull request apache/spark#13909
@viirya

How much overhead we can reduce with this?

Jun 25, 2016
viirya commented on pull request apache/spark#13778
@viirya

Yes. This PR adds it.

Jun 24, 2016
Jun 24, 2016
viirya commented on pull request apache/spark#13778
@viirya

python udf has no userClass. So regular handling of scala udf will be failed.

Jun 24, 2016
viirya commented on pull request apache/spark#13701
@viirya

@yhuai Ok. I will have it a try. But looks like I can only manually test it?

Jun 24, 2016
viirya commented on pull request apache/spark#13778
@viirya

also cc @yhuai @cloud-fan

Jun 24, 2016
viirya commented on pull request apache/spark#13701
@viirya

@yhuai As I mentioned in the description, I am not sure if we can manipulate row groups as we want, but I have manually tested it to show the actua…

Jun 24, 2016
viirya commented on pull request apache/spark#13701
@viirya

We use metadata in merged schema to mark the optional field (not existing in all partitions), the metadata is lost after resolving. If we don't add…

Jun 24, 2016
viirya commented on pull request apache/spark#13701
@viirya

retest this please.

Jun 24, 2016
viirya commented on pull request apache/spark#13701
@viirya

ping @liancheng @yhuai again...

Jun 24, 2016
@viirya
  • @nezihyigitbasi 63470af
    [SPARK-15782][YARN] Fix spark.jars and spark.yarn.dist.jars handling
  • @dhruve f1bf0d2
    [SPARK-15966][DOC] Add closing tag to fix rendering issue for Spark m…
  • 131 more commits »
Jun 24, 2016
viirya commented on pull request apache/spark#13778
@viirya

ping @vlad17 @davies @liancheng Any thing else?

Jun 23, 2016
@viirya
  • @viirya 66ab632
    Avoid creating String in getUTF8String.
Jun 22, 2016
viirya commented on pull request apache/spark#13775
@viirya

retest this please.

Jun 22, 2016
@viirya
Jun 22, 2016
@viirya
Jun 22, 2016
@viirya
Jun 22, 2016
viirya commented on pull request apache/spark#13778
@viirya

This test only fails when using show().

Jun 21, 2016
viirya commented on pull request apache/spark#13778
@viirya

@mengxr Although UDTs are private APIs, but as you see from the example, the users can define user classes and corresponding UDTs in Python that wi…

Jun 21, 2016
@viirya
Jun 21, 2016
viirya commented on pull request apache/spark#13772
@viirya

@yhuai What case you meant?

Jun 21, 2016
viirya commented on pull request apache/spark#13775
@viirya

@hvanhovell @rxin I've updated the benchmark. Please let me know if this time it is appropriate. Thanks!

Jun 20, 2016
viirya commented on pull request apache/spark#13778
@viirya

@vlad17 Thanks! I will look into that issue.

Jun 20, 2016
viirya commented on pull request apache/spark#13778
@viirya

Scala UDT is already cover by deserializerFor.

Jun 20, 2016
viirya commented on pull request apache/spark#13701
@viirya

@liancheng @yhuai How about this? Is it ready to merge? Thanks!

Something went wrong with that request. Please try again.