HIVE-26939 : Changed placement policy from NODE to node #3941
Conversation
…3.4 to parse correctly using Jackson 2.12.7
|
@abstractdog @zabetak Can you please review this. |
|
Kudos, SonarCloud Quality Gate passed!
|
|
thanks for this patch @amanraj2520, just for clarity's sake, what kind of tests are affected by this issue if we upgrade to hadoop 3.3.4? |
|
Hi @abstractdog Actually this error does not come up in a test until we manually bring up a cluster with Hive master and Hadoop 3.3.4. This error did not happen with hadoop 3.1.1 since at that time Hadoop was using Jackson 2.7.8 for deserializing this placement_policy yarnfile. Jackson 3.7.8 was able to deserialize it from "NODE" to "node" since it was case insensitive. Thus is found the [node] enum in the Hadoop PlacementScope class which in turn brought up the AM successfully. But Jackson 2.12.7 (Hadoop 3.3.4) is doing case sensitive parsing thereby not finding the relevant enum. Either we have to disable this configuration from jackson side 2.3/com/fasterxml/jackson/databind/MapperFeature.html#ACCEPT_CASE_INSENSITIVE_ENUMS or from Hive side we will have to send "node" as the placement policy. We can discuss on this Lazlo if you have doubts. |
|
Also I think we should backport it to 4.0 Hive release if this looks good. |
|
Hi @abstractdog If you see Hadoop 3.3.1 uses 2.10.x databind, in which I think this issue will also come while bringing up LLAP AM. Since I think jackson has modified this case-sensitive comparison from 2.9. Anyhow it is better to change it to node since Hadoop expects it as [node,rack] |
|
@abstractdog Can you please review this |
1 similar comment
|
@abstractdog Can you please review this |
|
one more question @amanraj2520: isn't the same issue happening with "type": "ANTI_AFFINITY"? |
|
Hi @abstractdog I do not think so. Because I was able to bring up the Hive LLAP with only change from NODE to node. But still will check this. |
|
Hi @abstractdog got the reason why ANTI_AFFINITY does not matter. If you see Hadoop 3.3.4 is using ANTI_AFFINITY in Upper Case in the PlacementType.class. Attaching the snippet here : Therefore it does not affect the jackson deserialization. |
|
The basic issue with Jackson 2.12.7 is that it does cases sensitive parsing so the request initiator(Hive) and receiver(Hadoop) should match cases. |
…3.4 to parse correctly using Jackson 2.12.7 (apache#3941) (Aman Raj reviewed by Laszlo Bodor)
…3.4 to parse correctly using Jackson 2.12.7 (apache#3941) (Aman Raj reviewed by Laszlo Bodor)
JIRA link : https://issues.apache.org/jira/browse/HIVE-26939
This is because llap_server uses templates.py yarnfile to bringup the LLAP Application Master which has the placement_policy as "NODE". Hadoop has started using 2.12.7 jackson version which has issues in parsing NODE to either [node, rack]. Therefore it is not able to cast the placement policy correctly.
Resolution : We will have to send placement_policy as "node" from the templates.py of Hive Llap Server module.
What changes were proposed in this pull request?
Why are the changes needed?
Does this PR introduce any user-facing change?
How was this patch tested?