Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-10520][SQL] Allow average out of DateType #28754

Closed
wants to merge 5 commits into from

Commits on Jun 12, 2020

  1. [SPARK-10520][SQL] Allow average out of DateType

    This allows to make an average out of date types.
    
    Under the hood we take an average of the days since
    epoch, and convert it to a date again. This requires
    the date object to be casted to a double to perform
    the average.
    
    Error in invokeJava(isStatic = FALSE, objId$id, methodName, ...) :
      org.apache.spark.sql.AnalysisException: cannot resolve 'avg(date)' due to data type mismatch: function average requires numeric types, not DateType;
    	at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
    	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:61)
    	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:53)
    	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:293)
    	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:293)
    	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:51)
    	at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:292)
    	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$5.apply(TreeNode.scala:290)
    	at org.apache.spark.sql.
    Fokko committed Jun 12, 2020
    Configuration menu
    Copy the full SHA
    020e701 View commit details
    Browse the repository at this point in the history
  2. Remove the timezone conversions

    Fokko committed Jun 12, 2020
    Configuration menu
    Copy the full SHA
    533dd8d View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2020

  1. Configuration menu
    Copy the full SHA
    9fba69e View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2020

  1. We can cast a double to a date

    This is days since epoch
    Fokko committed Jun 28, 2020
    Configuration menu
    Copy the full SHA
    bbf72c4 View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2020

  1. Configuration menu
    Copy the full SHA
    f499202 View commit details
    Browse the repository at this point in the history