-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix pyspark on spark3.0 #512
Conversation
RustRw
commented
Nov 19, 2020
- fix pyspark about py4j security
- fix DateTimeUtils miss stringToTime methed on spark3.0
@@ -65,6 +67,21 @@ class SparkPythonExecutor(val sc: SparkContext, val sqlContext: SQLContext,sess | |||
private[executors] var engineExecutorContext: EngineExecutorContext = _ | |||
private val lineOutputStream = new RsOutputStream | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
生成一个token
case _: TimestampType => new Timestamp(Try(timestampFormat.parse(value).getTime).getOrElse(DateTimeUtils.stringToTime(value).getTime * 1000L)) | ||
case _: DateType => new Date(Try(dateFormatP.parse(value).getTime).getOrElse(DateTimeUtils.stringToTime(value).getTime)) | ||
case _: TimestampType => new Timestamp(Try(timestampFormat.parse(value).getTime).getOrElse(stringToTime(value).getTime * 1000L)) | ||
case _: DateType => new Date(Try(dateFormatP.parse(value).getTime).getOrElse(stringToTime(value).getTime)) | ||
case _: StringType => value.replaceAll("\n|\t", " ") | ||
case t => throw new RuntimeException(s"Unsupported cast from $value to $t") | ||
} | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
针对 spark3.0 DataTimeUtils stringToTime方法并不存在了
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!