What‘s the difference between spark.rss.storage.type and rss.storage.type?

[xunxunmimi5577]

No description provided.

[jerqi]

spark.rss.storage.type is client's configuration.rss.storage.type is server's configuration.

[xunxunmimi5577]

Both parameters affect the storage mode of ShuffleServer？ Then，which one has a higher priority?

[jerqi]

Both parameters affect the storage mode of ShuffleServer？ Then，which one has a higher priority?

Only rss.storage.type affects.

[xunxunmimi5577]

What is spark.rss.storage.type used for？

[jerqi]

What is spark.rss.storage.type used for？

client

[xunxunmimi5577]

Does the client also can store data to a local disk or HDFS?

[jerqi]

Does the client also can store data to a local disk or HDFS?

Client need to know where to read data.

[xunxunmimi5577]

So these two parameters need be set to the same?

[jerqi]

So these two parameters need be set to the same?

Yes

[xunxunmimi5577]

ok,thanks

[xunxunmimi5577]

I have a problem,when I set storageType = HDFS, Error: Blocks read Inconsistent: Expected 61 Blocks, actual 0 Blocks
The version I'm using is 0.2.0

[xunxunmimi5577]

The configuration items are as follows
rss.rpc.server.port 19969
rss.jetty.http.port 19968
rss.storage.basePath /rss-0.2.0/shuffle-data1,/rss-0.2.0/shuffle-data2
rss.storage.type HDFS
rss.coordinator.quorum xxx:19999,xxx:19999,xxx:19999
rss.server.buffer.capacity 4gb
rss.server.buffer.spill.threshold 2gb
rss.server.partition.buffer.size 15mb
rss.server.read.buffer.capacity 2gb
rss.server.flush.thread.alive 5
rss.server.flush.threadPool.size 10
rss.server.health.check.enable false

#multistorage config
rss.server.multistorage.enable true
rss.server.uploader.enable true
rss.server.uploader.base.path hdfs:/firestorm/shuffle-data
rss.server.uploader.thread.number 3
rss.server.disk.capacity 5115506975

Spark client
spark.shuffle.manager org.apache.spark.shuffle.RssShuffleManager
spark.rss.coordinator.quorum xxx:19999,xxx:19999,xxx:19999
spark.rss.storage.type HDFS

[jerqi]

We don't recommend the storageType HDFS, you would better to use MEMORY_LOCALE.

[jerqi]

Storage HDFS will be removed soon.

[xunxunmimi5577]

Is there some problems with HDFS storage mode, so you want to remove it?

[xunxunmimi5577]

In terms of the results of my tests, only when I set storage type = LOCALFILE_HDFS that the shuffle data will be written to HDFS.
But I also find, in this case，if I delete local data, the same exception 'Blocks read Inconsistent' is reported, so i want to figure out if the function of reading shuffle data from HDFS is not sound enough so far.
Thanks for your time！

[jerqi]

Is there some problems with HDFS storage mode, so you want to remove it?

HDFS mode have poor performance. HDFS can work correctly.

[xunxunmimi5577]

Is there some problems with HDFS storage mode, so you want to remove it?

HDFS mode have poor performance. HDFS can work correctly.

Yes, you're right. I've got some configurations set incorrectly.