WIP - Hypercore Clone & Sync - With spawn/channels #21
Conversation
…low it executing each loop
|
Clones a metadata and content feeds from a hypercore-daemon peer given a hash and ip:port https://gist.github.com/bltavares/f37af663cb6f9fb4bd30ccabb1453a2a |
bltavares
changed the title
…e-protocol does not suport dns anymore
…e while we replicate in the background
|
This design has been validated to work with a daemon running replication in the background:
The relevant line is the https://github.com/bltavares/colmeia/pull/21/files#diff-60f4d526c84e9c5bcfc049b851ba7891R69-R89 That is quite promising, and I think this is proving to be a better design that #20 Now we have to do some cleanup and try to find a better way to manage the internal state machine, which is the core of the replication logic, without having a 300-line long match. I would also like to test if the service would be able to provide a single listener for all background tasks, so we could have multiple hyperdrives being served from a single hyperstack |
| .expect("Could not start hyperdrive on the stack"); | ||
| hyperstack.with_discovery(hyperstack.lan()); | ||
|
|
||
| let job = task::spawn(hyperstack.replicate()); |
There was a problem hiding this comment.
We can now replicate the storage info in the background, while still holding a reference to the driver to use it on another task.
| async fn main() -> Result<(), std::io::Error> { | ||
| env_logger::from_env(env_logger::Env::default().default_filter_or("info")).init(); | ||
|
|
||
| let key = name(); |
There was a problem hiding this comment.
Todo: use a better arguments parser with help messages
colmeiad/src/main.rs
Outdated
|
|
||
| let content = match &driver.content { | ||
| Some(content) => { | ||
| let len = content.read().await.len(); |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
|
||
| async fn get_info<Storage>(req: Request<State<Storage>>) -> tide::Result<tide::Response> | ||
| where | ||
| Storage: random_access_storage::RandomAccess<Error = Box<dyn std::error::Error + Send + Sync>> |
There was a problem hiding this comment.
TODO: Investigate datrs/hypercore#109 impact here - likely would be simpler to use a dyn Storage
| .await; | ||
|
|
||
| match event { | ||
| HyperdriveEvents::Client(Ok(proto::Event::DiscoveryKey(message))) => { |
There was a problem hiding this comment.
TODO: Can we write a simpler fn that is not 100s line long?
| hyperdrive: Arc<RwLock<Hyperdrive<Storage>>>, | ||
| ) where | ||
| C: AsyncRead + AsyncWrite + Send + Unpin + Clone + 'static, | ||
| Storage: random_access_storage::RandomAccess<Error = Box<dyn std::error::Error + Send + Sync>> |
There was a problem hiding this comment.
TODO: datrs/hypercore#109
colmeia-hypercore/src/network.rs
Outdated
| let mut remote_length = 0; | ||
|
|
||
| while let Some(message) = channel.next().await { | ||
| match message { |
There was a problem hiding this comment.
TODO: Can we have a fn that is not 100s line long?
colmeia-hypercore/src/lib.rs
Outdated
| mod schema; | ||
| mod utils; | ||
|
|
||
| pub use hyperdrive::*; |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
colmeia-hypercore/src/network.rs
Outdated
| } | ||
| } | ||
| HyperdriveEvents::Client(Err(_)) => { | ||
| // TODO delete this branch |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
To compare the approach with #20, this PR rewrites the whole architecture to spawn background tasks and communicate data upstream using channels instead of build adapters around streams.
Closes #20