layout | displayTitle | title | description | license |
---|---|---|---|---|
page |
Deploy Tez Client Plugin & Configurations |
Deploy Tez Client Plugin & Configurations |
Deploy Tez Client Plugin & Configurations |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
|
- Append client jar to package which is set by 'tez.lib.uris'.
In production mode, you can append client jar (rss-client-tez-XXXXX-shaded.jar) to package which is set by 'tez.lib.uris'.
In development mode, you can append client jar (rss-client-tez-XXXXX-shaded.jar) to HADOOP_CLASSPATH.
- Update tez-site.xml to enable Uniffle.
Property Name | Default | Description |
---|---|---|
tez.am.launch.cmd-opts | -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC org.apache.tez.dag.app.RssDAGAppMaster | enable remote shuffle service |
tez.rss.coordinator.quorum | coordinatorIp1:19999,coordinatorIp2:19999 | coordinator address |
Note that the RssDAGAppMaster will automatically disable slow start (i.e., tez.shuffle-vertex-manager.min-src-fraction=1
, tez.shuffle-vertex-manager.max-src-fraction=1
).
Property Name | Default | Description |
---|---|---|
tez.rss.avoid.recompute.succeeded.task | false | Whether to avoid recompute succeeded task when node is unhealthy or black-listed |
tez.rss.client.max.buffer.size | 3k | The max buffer size in map side. Control the size of each segment(WrappedBuffer) in the buffer. |
tez.rss.client.batch.trigger.num | 50 | The max batch of buffers to send data in map side. Affect the number of blocks sent to the server in each batch, and may affect rss_worker_used_buffer_size |
tez.rss.client.send.thread.num | 5 | The thread pool size for the client to send data to the server. |
tez.shuffle.mode | remote | Use Remote Shuffle if the value is set to 'remote' or use default config value, or set 'local' to use local shuffle when needs to fall back. |
In cloud environment, VM may have very limited disk space and performance. This experimental feature allows reduce tasks to spill data to remote storage (e.g., hdfs)
Property Name | Default | Description |
---|---|---|
tez.rss.reduce.remote.spill.enable | false | Whether to use remote spill |
tez.rss.remote.spill.storage.path | The remote spill path | |
tez.rss.reduce.remote.spill.replication | 1 | The replication number to spill data to Hadoop FS |
tez.rss.reduce.remote.spill.retries | 5 | The retry number to spill data to Hadoop FS |
Notice: this feature requires the MEMORY_LOCAL_HADOOP mode.