Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-36179][SQL] Support TimestampNTZType in SparkGetColumnsOperation #33393

Closed
wants to merge 5 commits into from
Closed

[SPARK-36179][SQL] Support TimestampNTZType in SparkGetColumnsOperation #33393

wants to merge 5 commits into from

Conversation

yaooqinn
Copy link
Member

What changes were proposed in this pull request?

Support TimestampNTZType in SparkGetColumnsOperation

Why are the changes needed?

TimestampNTZType coverage

Does this PR introduce any user-facing change?

yes, jdbc end-users will be aware of TimestampNTZType

How was this patch tested?

add new test

@github-actions github-actions bot added the SQL label Jul 16, 2021
@yaooqinn
Copy link
Member Author

cc @cloud-fan

@SparkQA
Copy link

SparkQA commented Jul 16, 2021

Test build #141153 has finished for PR 33393 at commit f713226.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jul 16, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45665/

@SparkQA
Copy link

SparkQA commented Jul 16, 2021

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45665/

@cloud-fan
Copy link
Contributor

The github action error seems not related to this PR

starting mypy test...
mypy checks failed:
python/pyspark/mllib/tree.pyi:29: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/tree.pyi:38: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:34: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:42: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:48: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:54: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:76: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:124: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:165: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/clustering.pyi:45: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/clustering.pyi:72: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/classification.pyi:39: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/classification.pyi:52: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
Found 13 errors in 4 files (checked 314 source files)

cc @HyukjinKwon

@SparkQA
Copy link

SparkQA commented Jul 19, 2021

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45782/

@SparkQA
Copy link

SparkQA commented Jul 19, 2021

Test build #141269 has finished for PR 33393 at commit fe990c6.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • public class ShuffleChecksumHelper
  • class MutableCheckedOutputStream(out: OutputStream) extends OutputStream
  • case class ShuffleChecksumBlockId(shuffleId: Int, mapId: Long, reduceId: Int) extends BlockId
  • class RemoteBlockDownloadFileManager(
  • case class SessionWindow(timeColumn: Expression, gapDuration: Long) extends UnaryExpression
  • case class LocalTimestamp(timeZoneId: Option[String] = None) extends LeafExpression
  • case class GetTimestamp(
  • case class ParseToTimestampLTZ(
  • case class ParseToTimestamp(
  • case class MakeTimestampNTZ(
  • case class MakeTimestampLTZ(
  • case class DomainJoin(
  • static class IntegerUpdater implements ParquetVectorUpdater
  • protected abstract class ConnectionProviderBase extends Logging
  • class MergingSortWithSessionWindowStateIterator(
  • trait HDFSBackedStateStoreMap
  • class NoPrefixHDFSBackedStateStoreMap extends HDFSBackedStateStoreMap
  • class PrefixScannableHDFSBackedStateStoreMap(
  • class HDFSBackedReadStateStore(val version: Long, map: HDFSBackedStateStoreMap)
  • class HDFSBackedStateStore(val version: Long, mapToUpdate: HDFSBackedStateStoreMap)
  • sealed trait RocksDBStateEncoder
  • class PrefixKeyScanStateEncoder(
  • class NoPrefixKeyStateEncoder(keySchema: StructType, valueSchema: StructType)
  • sealed trait StreamingSessionWindowStateManager extends Serializable
  • class StreamingSessionWindowStateManagerImplV1(
  • class StreamingSessionWindowHelper(sessionExpression: Attribute, inputSchema: Seq[Attribute])
  • case class SessionWindowStateStoreRestoreExec(
  • case class SessionWindowStateStoreSaveExec(

@HyukjinKwon
Copy link
Member

Yeah we can ignore that failure for now.

yaooqinn and others added 3 commits July 20, 2021 08:45
@yaooqinn @HyukjinKwon
Update sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive…
86fce93 
…/thriftserver/SparkMetadataOperationSuite.scala

Co-authored-by: Hyukjin Kwon <gurwls223@gmail.com>
@yaooqinn @HyukjinKwon
Update sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive…
356959c 
…/thriftserver/SparkMetadataOperationSuite.scala

Co-authored-by: Hyukjin Kwon <gurwls223@gmail.com>
@yaooqinn @HyukjinKwon
Update sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive…
f8c382b 
…/thriftserver/SparkMetadataOperationSuite.scala

Co-authored-by: Hyukjin Kwon <gurwls223@gmail.com>
@HyukjinKwon
Copy link
Member

Merged to master and branch-3.2

HyukjinKwon pushed a commit that referenced this pull request Jul 20, 2021
@yaooqinn @HyukjinKwon
[SPARK-36179][SQL] Support TimestampNTZType in SparkGetColumnsOperation
782dc9a 
### What changes were proposed in this pull request?

Support TimestampNTZType in SparkGetColumnsOperation

### Why are the changes needed?

TimestampNTZType coverage

### Does this PR introduce _any_ user-facing change?

yes, jdbc end-users will be aware of TimestampNTZType

### How was this patch tested?

add new test

Closes #33393 from yaooqinn/SPARK-36179.

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 0c76fb9)
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
@yaooqinn yaooqinn deleted the SPARK-36179 branch July 22, 2021 07:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@HyukjinKwon HyukjinKwon HyukjinKwon approved these changes

@cloud-fan cloud-fan cloud-fan approved these changes

Assignees
No one assigned
Labels
SQL
Projects
None yet
Milestone
No milestone
4 participants
@yaooqinn @SparkQA @cloud-fan @HyukjinKwon