[Bug] Paimon Table produce more reduce task for same data volume

[JingsongLi]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

0.4

Compute Engine

hive

Minimal reproduce step

Compare the paimon table and hive table with the same statement and data amount using the hive client:
painmon table:

hive table：

What doesn't meet your expectations?

read paimon table should have lower reducer number.

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[calvinjiang]

I'd like to fix this issue.

[JingsongLi]

I'd like to fix this issue.

How to?

[wg1026688210]

hi~ @JingsongLi Is it because there are more files in the Paimon table. Is it effective to reduce the number of reduce tasks, If we set number of the upstream map by mapred.map.tasks ,which reduce the number of shuffle files.

[JingsongLi]

hi~ @JingsongLi Is it because there are more files in the Paimon table. Is it effective to reduce the number of reduce tasks, If we set number of the upstream map by mapred.map.tasks ,which reduce the number of shuffle files.

We should figure out what is the mechanism of the task number inference in Hive, and try to work around.