[SPARK-22635][SQL][ORC] FileNotFoundException while reading ORC files containing special characters #19844
Conversation
… containing special characters
|
@dongjoon-hyun @gatorsmile you helped reviewing SPARK-22146. Might you please help reviewing this too? Sorry for pinging you directly. |
| @@ -59,8 +59,9 @@ class OrcFileFormat extends FileFormat with DataSourceRegister with Serializable | |||
| sparkSession: SparkSession, | |||
| options: Map[String, String], | |||
| files: Seq[FileStatus]): Option[StructType] = { | |||
| val fileNames = files.map(_.getPath.toString) | |||
There was a problem hiding this comment.
not a big deal but just curious - why did you name it fileNames BTW? I thought it's going to be just like paths.
There was a problem hiding this comment.
because they are String and not Paths. Actually this change is not needed, I changed it only because it was easier for me while debugging. I can put it back as it was before.
There was a problem hiding this comment.
I mean it's not the name of a file though.
There was a problem hiding this comment.
yes, I reverted the change, thanks.
|
Test build #84292 has finished for PR 19844 at commit
|
|
Test build #84304 has finished for PR 19844 at commit
|
|
ping, @cloud-fan , too. |
|
Merged to master. |
|
thanks @HyukjinKwon , may I kindly ask you to backport also to branch-2.2? Thanks. |
|
I think SPARK-22146 is not backported though? |
|
Ah, it was mistake of fixed version in the JIRA. Sure, will push it too. |
… containing special characters ## What changes were proposed in this pull request? SPARK-22146 fix the FileNotFoundException issue only for the `inferSchema` method, ie. only for the schema inference, but it doesn't fix the problem when actually reading the data. Thus nearly the same exception happens when someone tries to use the data. This PR covers fixing the problem also there. ## How was this patch tested? enhanced UT Author: Marco Gaido <mgaido@hortonworks.com> Closes #19844 from mgaido91/SPARK-22635.
|
Merged to branch-2.2 too. |
|
thank you @HyukjinKwon |
… containing special characters ## What changes were proposed in this pull request? SPARK-22146 fix the FileNotFoundException issue only for the `inferSchema` method, ie. only for the schema inference, but it doesn't fix the problem when actually reading the data. Thus nearly the same exception happens when someone tries to use the data. This PR covers fixing the problem also there. ## How was this patch tested? enhanced UT Author: Marco Gaido <mgaido@hortonworks.com> Closes apache#19844 from mgaido91/SPARK-22635.
What changes were proposed in this pull request?
SPARK-22146 fix the FileNotFoundException issue only for the
inferSchemamethod, ie. only for the schema inference, but it doesn't fix the problem when actually reading the data. Thus nearly the same exception happens when someone tries to use the data. This PR covers fixing the problem also there.How was this patch tested?
enhanced UT