[HUDI-3008] Fixing HoodieFileIndex partition column parsing for nested fields #4308
Conversation
|
Can we add tests please |
added |
| * @param structField | ||
| * @return map of ( columns names -> StructField ) | ||
| */ | ||
| private def convertStructToMap(structField : Either[StructField,StructType]) : Map[String,StructField] = { |
There was a problem hiding this comment.
@xiarixiaoyao : can you review this patch please.
There was a problem hiding this comment.
nit: can we name this method "generateNameFieldMap" or "fetchNameFieldMap"
There was a problem hiding this comment.
generateNameFieldMap
Done
|
@hudi-bot run azure |
There was a problem hiding this comment.
LGTM except for one minor comment. Lets wait for a day to hear from @xiarixiaoyao as well.
nsivabalan
changed the title
nsivabalan
changed the title
| } | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
pls notice the code style of scala：
structField : Either[StructField,StructType] => structField: Either[StructField, StructType]
Map[String,StructField] => Map[String, StructField]
case (key :String , sf:StructField) => (field.name + "." + key , sf ) => case (key: String, sf: StructField) => (field.name + "." + key, sf )
|
LGTM， |
Done |
| assert(fileIndex.getAllQueryPartitionPaths.get(0).partitionPath.equals("c")) | ||
| } | ||
|
|
||
|
|
There was a problem hiding this comment.
nit: remove the empty line here?
|
@hudi-bot run azure |
|
@harsh1231 : Can you rebase with latest master. |
harsh1231
force-pushed
the
HUDI-3008
branch
2 times, most recently
from
e890932
to
168fb8f
Compare
Done |
|
@xiarixiaoyao can you please take a look and either approve the PR or suggest next steps? |
|
@hudi-bot run azure |
|
@harsh1231， Thank you very much for your contribution. I have a little suggestion： pls pay attention to scala's code style, once fixed, we can merge this pr. |
| * @return map of ( columns names -> StructField ) | ||
| */ | ||
| private def generateNameFieldMap(structField : Either[StructField, StructType]) : Map[String, StructField] = { | ||
| structField match { |
There was a problem hiding this comment.
Please delete the space before ‘:’
private def generateNameFieldMap(structField : Either[StructField, StructType]) : Map[String, StructField] =>
private def generateNameFieldMap(structField: Either[StructField, StructType]): Map[String, StructField]
There was a problem hiding this comment.
Done, thanks for pointing out code style of scala.
|
@hudi-bot run azur |
|
@hudi-bot run azure |
1 similar comment
|
@hudi-bot run azure |
Tips
What is the purpose of the pull request
a.b.c.which is struct in nature , loading hoodie table through spark was givingCannot find column: 'a.b.c' in the schemaRun Command :
spark.read.format("hudi").load("<table_path>").show()This PR fixes above error by traversing schema struct recursively.
Brief change log
(for example:)
Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)
Committer checklist
Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.