[Bug] Incorrect local disk size limitation when colocating with Yarn nodemanager

[zuston]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

When the shuffle server of uniffle colocates with the Yarn nodemanager and using MEMORY_LOCALFILE_HDFS, I found the exception occurs, like this: Unexpected crc value for blockId xxxxxxxxx . (Finally, I found this is caused by the partial data into file when meeting the no-space left exception)

From the corresponding shuffle-server, I found the disk is full.
But why the disk full caused the unexpected crc? I give two possible reasons

1. The flush event is triggered to write the local disk due to the failure of storage.canWrite()
2. And then the partial data of one event is written to the file of this disk. The another part data for this event has not been written due to no-space left exception. This makes in-consistency

I want to explain more about stoarge.canWrite() . Actually, this part logic is correct for exclusive resource deployment for uniffle shuffle-server. But when colocating with other service, this method is incorrect. Because the other service may occupy lots of disk size, which will make the disk limit invalid.

Affects Version(s)

master

Uniffle Server Log Output

No response

Uniffle Engine Log Output

No response

Uniffle Server Configurations

No response

Uniffle Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[roryqi]

This issue is similar to your previous issue. @awdavidson Do you have any progress? Do you have another suggestion?

[zuston]

This issue is similar to your previous issue. @awdavidson Do you have any progress? Do you have another suggestion?

Oh yes. Let @awdavidson to fix this if he is interested

[zuston]

BTW, do we support atomic writing for one event? @jerqi WDYT

[awdavidson]

@jerqi I sounds similar except the storage type and exceptions are different. I was using MEMORY_LOCALFILE and noticed that LocalStorageMeta disk size was not correctly updated so when disk was full uniffle would keep trying writing shuffle data. I have a simple fix for this, will look at contributing today. Sorry been busy the past week

[roryqi]

BTW, do we support atomic writing for one event? @jerqi WDYT

I think we don't need. Is it necessary? Usually atomic write operation will bring more cost. I'm worried whether it affected our performance and code complexity。

[awdavidson]

@jerqi @zuston this is the change required to ensure all events update LocalStorageMeta disk size #902

[awdavidson]

@zuston are we good to close this issue now?

[zuston]

@zuston are we good to close this issue now?

I think this is another problem when colocating with other service.

[awdavidson]

@zuston are we good to close this issue now?

I think this is another problem when colocating with other service.

Yes you are correct. Uniffle is not aware of the actual available space