Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better example of simple-encoders/decoder #29

Closed
spikebike opened this issue Jul 23, 2016 · 3 comments
Closed

Better example of simple-encoders/decoder #29

spikebike opened this issue Jul 23, 2016 · 3 comments

Comments

@spikebike
Copy link

spikebike commented Jul 23, 2016

I was tinkering with the examples and as a bit surprised that a single bit error can corrupt the decoding. The README does mention this "If values have changed in a shard, it cannot be reconstructed".

I was thinking it might be useful to have an example closer to a real world use case.

The example README does mention hashing the shards so you can delete the corrupt ones. But in the default example (4 shards + 2 parity) even with a hash of each shard, 3 single bit errors could kill a decoding.

So I was thinking of adding a sha256 per 64KB block. That way any bit corruptions would be recognized and you could use the other 5 shards for that particular block.

I wrote a little simulator to add errors and I believe that on average a 2GB file encoded with the defaults (4 shards + 2 parity shards) would survive (on average) between 280 and 1200 64KB errors before any part of the file is corrupted. Quite a bit better than 3 bit errors. Storage overhead would be something like 0.05%. Additionally even with 8192 errors 95% of the file would be recoverable.

Would you accept a pull request to add sha256 block checksums to your simple encoder/decoder?

@klauspost
Copy link
Owner

What do you think is missing in the current ones?

@spikebike
Copy link
Author

spikebike commented Jul 23, 2016

Original issue edited.

@spikebike spikebike changed the title Better example of simple-ensoders/decoder? Better example of simple-encoders/decoder Jul 23, 2016
@klauspost
Copy link
Owner

@spikebike Feel free to open a PR if you still have the better example.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants