Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(iroh-bytes): bring back validation #2107

Merged
merged 21 commits into from
Mar 22, 2024
Merged

feat(iroh-bytes): bring back validation #2107

merged 21 commits into from
Mar 22, 2024

Conversation

rklaehn
Copy link
Contributor

@rklaehn rklaehn commented Mar 21, 2024

Description

feat(iroh-bytes): bring back validation

This brings back validation, and also separates validation and consistency check and validation.

Rationale:

  • consistency check is highly dependent on the store impl. E.g. for an in memory impl it might not even make sense.
  • validate is much more expensive, but can be mostly implemented in a store impl independent way. It just needs access to the data.

consistency check output at very high verbosity:
(basically dumps the db and then validates each entry)
image

validate output:
(validates all complete files, lists ranges for all partial files)
image

Notes & open questions

  • Note: repairing by downgrading complete entries to partial entries is not yet implemented. It is a bit tricky.
  • Note: both are implemented as a RPC command for a running store, and in iroh doctor for a separate file.

Change checklist

  • Self-review.
  • Documentation updates if relevant.
  • Tests if relevant.

…a concrete type.

This is necessary to send progress senders via actor messages, but we also
probably don't want to use a concrete progress sender type in the actor
messges.
# Conflicts:
#	Cargo.lock
#	iroh-base/Cargo.toml
#	iroh-bytes/src/store/bao_file.rs
#	iroh-bytes/src/store/file.rs
#	iroh-bytes/src/store/file/tests.rs
#	iroh-bytes/src/store/mem.rs
#	iroh-bytes/src/store/readonly_mem.rs
BaoFileHandle is exactly the entry.
- separate fsck aka consistency check and validate
@rklaehn rklaehn marked this pull request as draft March 21, 2024 10:31
@rklaehn rklaehn marked this pull request as ready for review March 21, 2024 14:02
# Conflicts:
#	iroh-bytes/src/store/fs/validate.rs
#	iroh-cli/src/commands/doctor.rs
it is no longer used in validate
# Conflicts:
#	iroh-cli/src/commands/blob.rs
#	iroh/src/client/blobs.rs
#	iroh/src/node/rpc.rs
#	iroh/src/rpc_protocol.rs
@rklaehn rklaehn added this pull request to the merge queue Mar 22, 2024
Merged via the queue into main with commit 50b3e47 Mar 22, 2024
20 checks passed
@rklaehn rklaehn deleted the enhanced-validation branch March 22, 2024 14:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

None yet

2 participants