You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have searched in the issues and found no similar issues.
Describe the bug
big shuffle data will flush to hdfs. But shuffle can no read these data. The reason is that baseDir is missing.
org.apache.uniffle.common.exception.RssFetchFailedException: Failed to read shuffle data from WARM handler
at org.apache.uniffle.storage.handler.impl.ComposedClientReadHandler.readShuffleData(ComposedClientReadHandler.java:124)
at org.apache.uniffle.storage.handler.impl.ComposedClientReadHandler.readShuffleData(ComposedClientReadHandler.java:134)
at org.apache.uniffle.client.impl.ShuffleReadClientImpl.read(ShuffleReadClientImpl.java:348)
at org.apache.uniffle.client.impl.ShuffleReadClientImpl.readShuffleBlockData(ShuffleReadClientImpl.java:267)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.RssTezShuffleDataFetcher.copyFromRssServer(RssTezShuffleDataFetcher.java:144)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.RssTezShuffleDataFetcher.fetchAllRssBlocks(RssTezShuffleDataFetcher.java:129)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.RssTezShuffleDataFetcher.callInternal(RssTezShuffleDataFetcher.java:108)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.RssTezShuffleDataFetcher.callInternal(RssTezShuffleDataFetcher.java:38)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at org.apache.uniffle.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at org.apache.uniffle.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
at org.apache.uniffle.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.uniffle.common.exception.RssException: Can't get FileSystem for null/appattempt_1691045129773_0174_000001/1000001/5-5
at org.apache.uniffle.storage.handler.impl.HadoopClientReadHandler.init(HadoopClientReadHandler.java:129)
at org.apache.uniffle.storage.handler.impl.HadoopClientReadHandler.readShuffleData(HadoopClientReadHandler.java:199)
at org.apache.uniffle.storage.handler.impl.ComposedClientReadHandler.readShuffleData(ComposedClientReadHandler.java:113)
... 14 more
Affects Version(s)
master
Uniffle Server Log Output
No response
Uniffle Engine Log Output
No response
Uniffle Server Configurations
No response
Uniffle Engine Configurations
No response
Additional context
No response
Are you willing to submit PR?
Yes I am willing to submit a PR!
The text was updated successfully, but these errors were encountered:
zhengchenyu
added a commit
to zhengchenyu/incubator-uniffle
that referenced
this issue
Aug 8, 2023
…dfs (#1118)
### What changes were proposed in this pull request?
Apply remote storage configuration.
### Why are the changes needed?
Reduce does not load remote storage path. If shuffle data have flushed to remote storage, reduce can not read.
Fix: #1081
### How was this patch tested?
test in cluster and UT.
Code of Conduct
Search before asking
Describe the bug
big shuffle data will flush to hdfs. But shuffle can no read these data. The reason is that baseDir is missing.
Affects Version(s)
master
Uniffle Server Log Output
No response
Uniffle Engine Log Output
No response
Uniffle Server Configurations
No response
Uniffle Engine Configurations
No response
Additional context
No response
Are you willing to submit PR?
The text was updated successfully, but these errors were encountered: