HIVE-24485: Make the slow-start behavior tunable

[okumin]

What changes were proposed in this pull request?

Let Hive respect tez.shuffle-vertex-manager.min-src-fraction and tez.shuffle-vertex-manager.max-src-fraction when auto reducer parallelism is enabled. Currently, we don't have any methods to configure the parameters.

https://issues.apache.org/jira/browse/HIVE-24485

Why are the changes needed?

We sometimes want to tune the parameters so that Tez can finalize the number of reducers with a more accurate estimation. For example, in extreme cases where the previous stage is very skewed, we may want to configure both params at 1.0.

Does this PR introduce any user-facing change?

The behavior will change if a user configures tez.shuffle-vertex-manager.{min,max}-src-fraction.

How was this patch tested?

zookage@client-node-0:~$ beeline -e '
> SET hive.tez.auto.reducer.parallelism=true;
> SET hive.tez.min.partition.factor=1.0; -- enforce auto-parallelism
> SET tez.shuffle-vertex-manager.min-src-fraction=0.55;
> SET tez.shuffle-vertex-manager.max-src-fraction=0.95;
> CREATE TABLE mofu (name string);
> INSERT INTO mofu (name) VALUES ('12345');
> SELECT name, count(*) FROM mofu GROUP BY name;
> '

2023-06-11 15:50:57,287 [INFO] [Dispatcher thread {Central}] |vertexmanager.ShuffleVertexManagerBase|: Settings minFrac: 0.55 maxFrac: 0.95 auto: true desiredTaskIput: 256000000

[okumin]

I think the first commit is the simplest.
However, Gopal suggested I should move to the current style while the previous PR is open.
https://issues.apache.org/jira/browse/HIVE-24485?focusedCommentId=17278522&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17278522

I personally don't have very strong opinions about how to support the params.

[abstractdog]

@okumin : thanks for working on this, I believe the latest patch with Gopal's suggestion looks good, and I feel we can support all (currently 5) properties like this:
https://github.com/apache/tez/blob/ccad71dabceb850b58ff38e77fd206e68f62ce4b/tez-runtime-library/src/main/java/org/apache/tez/dag/library/vertexmanager/ShuffleVertexManager.java#L71-L110

[okumin]

@abstractdog Thanks for putting your opinion. Actually, all the other 3 params are already covered indirectly. The newly added 2 are only ones we can't control at all.

hive/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java

Lines 1657 to 1669 in b55f9c5

	pluginConf.setBoolean(
	ShuffleVertexManager.TEZ_SHUFFLE_VERTEX_MANAGER_ENABLE_AUTO_PARALLEL, true);
	pluginConf.setInt(ShuffleVertexManager.TEZ_SHUFFLE_VERTEX_MANAGER_MIN_TASK_PARALLELISM,
	edgeProp.getMinReducer());
	pluginConf.setLong(
	ShuffleVertexManager.TEZ_SHUFFLE_VERTEX_MANAGER_DESIRED_TASK_INPUT_SIZE,
	edgeProp.getInputSizePerReducer());
	pluginConf.setFloat(
	ShuffleVertexManager.TEZ_SHUFFLE_VERTEX_MANAGER_MIN_SRC_FRACTION,
	edgeProp.getMinSrcFraction());
	pluginConf.setFloat(
	ShuffleVertexManager.TEZ_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION,
	edgeProp.getMaxSrcFraction());

• tez.shuffle-vertex-manager.enable.auto-parallel: This is configured when hive.tez.auto.reducer.parallelism is enabled and the reduce vertex can be parallelized
• tez.shuffle-vertex-manager.desired-task-input-size: This is inherited from hive.exec.reducers.bytes.per.reducer
• tez.shuffle-vertex-manager.min-task-parallelism: This will be {estimated # of reducers} * hive.tez.min.partition.factor

If we support them, it means tez.shuffle-vertex-manager.* overwrites the 3 params. Is it preferable?
Another option is to introduce new params like hive.tez.auto.reducer.parallelism.min-src-fraction assuming tez.shuffle-vertex-manager.* is SPI-like params. It might be consistent with other params.

[abstractdog]

thanks for the detailed explanation @okumin, I prefer not changing the existing behavior regarding the other properties

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
3 Code Smells

No Coverage information
No Duplication information

[okumin]

Thanks for reviewing this PR!