Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Batch gossip #1143

Merged
merged 15 commits into from Dec 3, 2021
Merged

Batch gossip #1143

merged 15 commits into from Dec 3, 2021

Conversation

freesig
Copy link
Contributor

@freesig freesig commented Nov 29, 2021 •
edited

Summary

TODO:

  • CHANGELOG(s) updated with appropriate info
  • Just before pressing the merge button, ensure new entries to CHANGELOG(s) are still under the UNRELEASED heading

@freesig freesig marked this pull request as ready for review November 30, 2021 06:40
@freesig freesig requested review from neonphog and maackle and removed request for neonphog November 30, 2021 07:46
neonphog
neonphog reviewed Nov 30, 2021
View reviewed changes
crates/holochain_sqlite/src/sql/cell/schema.sql
@@ -144,7 +144,7 @@ CREATE INDEX IF NOT EXISTS DhtOp_validation_status_idx ON DhtOp ( validation_sta
CREATE INDEX IF NOT EXISTS DhtOp_authored_timestamp_idx ON DhtOp ( authored_timestamp );
CREATE INDEX IF NOT EXISTS DhtOp_storage_center_loc_idx ON DhtOp ( storage_center_loc );
CREATE INDEX IF NOT EXISTS DhtOp_header_hash_idx ON DhtOp ( header_hash );
-- CREATE INDEX DhtOp_basis_hash_idx ON DhtOp ( basis_hash );
CREATE INDEX IF NOT EXISTS DhtOp_basis_hash_idx ON DhtOp ( basis_hash );
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this the magic index we needed? : )

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep 🚀

neonphog
neonphog reviewed Nov 30, 2021
View reviewed changes
crates/kitsune_p2p/kitsune_p2p/src/gossip/sharded_gossip.rs
/// Is this the last chunk of ops to be sent in response
/// to the bloom filter that we're responding to?
finished.1: bool,
/// Ops that are missing from a bloom that you have sent.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are these docs supposed to go above the ops.0 field?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really, it's more about explaining the finished MissingOpsStatus

neonphog
neonphog reviewed Nov 30, 2021
View reviewed changes
crates/kitsune_p2p/types/src/config.rs
/// [Default: 16]
tx2_channel_count_per_connection: usize = 16,
/// [Default: 2]
tx2_channel_count_per_connection: usize = 2,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

neonphog
neonphog approved these changes Nov 30, 2021
View reviewed changes
Copy link
Contributor

@neonphog neonphog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

woot

maackle
maackle reviewed Dec 1, 2021
View reviewed changes
Copy link
Member

@maackle maackle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still working through this

crates/kitsune_p2p/kitsune_p2p/src/gossip/sharded_gossip.rs Outdated Show resolved Hide resolved
crates/kitsune_p2p/kitsune_p2p/src/gossip/sharded_gossip.rs Outdated Show resolved Hide resolved
crates/kitsune_p2p/kitsune_p2p/src/gossip/sharded_gossip/bloom.rs
Comment on lines +75 to +78
.try_take_while(|_| {
total_blooms += 1;
futures::future::ready(Ok(total_blooms <= Self::UPPER_BLOOM_BOUND))
})
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that different than just this?

Suggested change
.try_take_while(|_| {
total_blooms += 1;
futures::future::ready(Ok(total_blooms <= Self::UPPER_BLOOM_BOUND))
})
.take(Self::UPPER_BLOOM_BOUND)

Copy link
Contributor Author

@freesig freesig Dec 1, 2021 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately there's no take for a TryStream

crates/kitsune_p2p/kitsune_p2p/src/gossip/sharded_gossip/initiate.rs Outdated Show resolved Hide resolved
crates/kitsune_p2p/kitsune_p2p/src/gossip/sharded_gossip/ops.rs
)]),
}
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

checkpoint: this is as far as I got in one sitting

maackle
maackle approved these changes Dec 2, 2021
View reviewed changes
crates/kitsune_p2p/kitsune_p2p/src/gossip/sharded_gossip/ops.rs

let got_len = missing_ops.len();

// If there is less ops then missing hashes the call was batched.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because they are limited by the fetch ops call

crates/kitsune_p2p/kitsune_p2p/src/gossip/sharded_gossip/store.rs Outdated Show resolved Hide resolved
@freesig freesig merged commit e9f7fe5 into develop Dec 3, 2021
@freesig freesig deleted the batch-gossip branch December 3, 2021 02:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@neonphog neonphog neonphog approved these changes

@maackle maackle maackle approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@freesig @neonphog @maackle