[HUDI-6072] Fix NPE when upsert merger and null map or array

[parisni]

Change Logs

Fixes #8431

Impact

Describe any public API or user-facing feature change or any performance impact.

Risk level (write none, low medium or high below)

none

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change

• The config description must be updated if new configs are added or the default value of the configs are changed
• Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
  ticket number here and follow the instruction to make
  changes to the website.

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[parisni]

Tests based on the spark datasource api (similar to the eg in the OP) or do you have a better idea?

…

On April 13, 2023 6:35:17 AM UTC, Danny Chan ***@***.***> wrote: @danny0405 commented on this pull request. > - val newArrayData = createArrayData(newElementType, prevArrayData.numElements()) - val elementUpdater = new ArrayDataUpdater(newArrayData) - - var i = 0 - while (i < prevArray.length) { - val element = prevArray(i) - if (element == null) { - if (!containsNull) { - throw new HoodieException( - s"Array value at path '${fieldNameStack.asScala.mkString(".")}' is not allowed to be null") + if (value == null) { + fieldUpdater.setNullAt(ordinal) + } else { + val prevArrayData = value.asInstanceOf[ArrayData] + val prevArray = prevArrayData.toObjectArray(prevElementType) + Thanks for the contribution, can we add some test cases? -- Reply to this email directly or view it on GitHub: #8432 (review) You are receiving this because you authored the thread. Message ID: ***@***.***>

[danny0405]

Tests based on the spark datasource api (similar to the eg in the OP) or do you have a better idea?

Either data sorurce api or sql should be fine.

[parisni]

@danny0405 added stuff for tests, bug cannot setup intellij to debug/run tests locally (pain following the https://hudi.apache.org/contribute/developer-setup)

[danny0405]

@danny0405 added stuff for tests, bug cannot setup intellij to debug/run tests locally (pain following the https://hudi.apache.org/contribute/developer-setup)

Could we fix line 191 instead, that could fix all the data type with null values, covering also the atomic data types.

You can also use the mvn test cmd for testing.

[danny0405]

Thanks for the fix, I have reviewed and created a patch:
6072.patch.zip
You can rebase with the latest master, apply the patch and then force-push with the branch.

[danny0405]

@hudi-bot run azure

[parisni]

@danny0405 integrated your patch. Now I need to:

• confirm the test breaks without the change
• assert not throw npe

[danny0405]

@danny0405 integrated your patch. Now I need to:

• confirm the test breaks without the change
• assert not throw npe

Yeah, that makes sense, I run the 2 tests locally and they can pass.

[parisni]

@danny0405 added a commit to apply the exact same context as the issue. The previous tests did not fail w/o the patch.

[danny0405]

+1

[hudi-bot]

CI report:

• 8a67f29 UNKNOWN
• 22fb605 Azure: SUCCESS
• 2e3f72f Azure: PENDING

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build