How uniffle write data to hdfs

[xianjingfeng]

If rss.data.replica.write=2 and rss.storage.type=MEMORY_LOCALFILE_HDFS

1. What will it do if the local disk does not have enough space
2. Whether it will write two copies to hdfs

[xianjingfeng]

@jerqi @frankliee Can you answer? I'm going to change the logic of this section, because we need to write to HDFS when the local disk is full. But I'm not sure my understanding is correct, and i want to know the plan of community

[jerqi]

1.In our production environment, the space of disk is enough in most cases, and HDFS is only used to solve the problem that local disk cannot be written.So i think in this case, the performance will not bad. Should we add an option for users to choose? 2.Do you have any plan to optimize this logic?

1. If local disk can't be written, can the local disk be read? If local disk can't be read, it means that we lost the data. Although we can write data to hdfs, we also need to face the data lost problems.
2. I don't have similar plan.

[xianjingfeng]

1. Disk can be read, and when disk usage exceeds the water mark, it can't be written.

[jerqi]

If we encounter the disk full usage problem, we can use MEMORY_LOCALFILE_HDFS directly.

[xianjingfeng]

But the performance of writing HDFS is not good now, especially cross DC, because it will write two copies if rss.data.replica.write=2 and it is difficult to control how much percent is written to HDFS.

[jerqi]

But the performance of writing HDFS is not good now, especially cross DC, because it will write two copies if rss.data.replica.write=2 and it is difficult to control how much percent is written to HDFS.

In current implement, we choose big data to write to HDFS, the 75% of data will be written to HDFS. It has good performance. And it's ok for me when local disk is full, we choose to write HDFS. It's reasonable fallback mechanism.