Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(iroh): downloader #1420

Merged
merged 97 commits into from
Sep 13, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
97 commits
Select commit Hold shift + click to select a range
20da0ef
starting from scratch
divagant-martian Aug 22, 2023
03869bd
add download handle
divagant-martian Aug 22, 2023
ceed4fd
add sources and download info
divagant-martian Aug 22, 2023
aab7849
add response sender
divagant-martian Aug 23, 2023
3480f79
Merge branch 'main' into download-manager
divagant-martian Aug 23, 2023
313cba4
add fields for scheduling, code cancellation
divagant-martian Aug 23, 2023
031b59d
handle receiving a start download request
divagant-martian Aug 24, 2023
47fcf3a
improve docs
divagant-martian Aug 24, 2023
81c77ce
add collection parser and store
divagant-martian Aug 26, 2023
7a97b2c
Merge branch 'main' into download-manager
divagant-martian Aug 26, 2023
9227ea3
add docs
divagant-martian Aug 27, 2023
b8c916c
add pending count fn to Dialer
divagant-martian Aug 27, 2023
e8c1f99
add dialer, availability registry and concurrency limits
divagant-martian Aug 27, 2023
b08733d
add parts
divagant-martian Aug 28, 2023
325d1c9
remove unused source enum. delegated now to the availability_registry
divagant-martian Aug 28, 2023
370b436
add get_best_candidate
divagant-martian Aug 28, 2023
40664a8
dial selected best peer
divagant-martian Aug 29, 2023
aa2b3ef
add comments
divagant-martian Aug 29, 2023
82ab766
filling in random gaps
divagant-martian Aug 29, 2023
ba29846
Merge branch 'main' into download-manager
divagant-martian Aug 29, 2023
0e362d5
start adding errors based on fsm error management
divagant-martian Aug 29, 2023
386edae
remove progress stuff
divagant-martian Aug 30, 2023
3284fa9
do not add swear words in commit messages
divagant-martian Aug 30, 2023
e75402c
reworking active vs scheduled requests
divagant-martian Aug 30, 2023
e40ee09
rework pending and connected info plus retry count
divagant-martian Sep 1, 2023
0fb64b5
simplify cancelling a request
divagant-martian Sep 1, 2023
29a8068
rework get_peer_connection_for_download
divagant-martian Sep 1, 2023
05045a2
excluding peers is not necessary
divagant-martian Sep 1, 2023
e8f79bf
remove unused import
divagant-martian Sep 1, 2023
6d7f992
add retry constants
divagant-martian Sep 1, 2023
91ca199
add retry backoff calculation
divagant-martian Sep 1, 2023
0bab11c
remove obsolte code
divagant-martian Sep 1, 2023
1353179
handle requests ready for processing
divagant-martian Sep 1, 2023
bac6c0d
typos and docs
divagant-martian Sep 1, 2023
892245d
connect download
divagant-martian Sep 1, 2023
558d1b9
remove unused method
divagant-martian Sep 2, 2023
d28d1ea
typo
divagant-martian Sep 3, 2023
191a726
add on_download_completed
divagant-martian Sep 3, 2023
79d9fd2
lifetime issues
divagant-martian Sep 3, 2023
7427f00
banning via removing the connection
divagant-martian Sep 3, 2023
8673f79
prevent banned peers from taking all the connected peer slots
divagant-martian Sep 3, 2023
b046f37
add peer expiry
divagant-martian Sep 4, 2023
cda280e
extend to add and remove peers from the registry
divagant-martian Sep 4, 2023
ca380d1
add dummy registry
divagant-martian Sep 4, 2023
4def2ab
fix imports
divagant-martian Sep 4, 2023
d00a340
spawn service
divagant-martian Sep 4, 2023
6eacf27
not so hot hot-swaping
divagant-martian Sep 4, 2023
11388cd
Merge branch 'main' into download-manager
divagant-martian Sep 4, 2023
5ba80fe
hot-swaping in examples
divagant-martian Sep 4, 2023
c518f15
fix example
divagant-martian Sep 4, 2023
c9cbc39
fix some docs and imports
divagant-martian Sep 4, 2023
0f75cec
fix docs
divagant-martian Sep 5, 2023
2514acd
our lord clippy
divagant-martian Sep 5, 2023
c1b0b6e
more clippy
divagant-martian Sep 5, 2023
5b4b120
Merge branch 'main' into download-manager
divagant-martian Sep 5, 2023
c6dcd3b
allow to add initial peers at the same time with the download
divagant-martian Sep 5, 2023
d3535c7
debug failing test
divagant-martian Sep 5, 2023
35416d0
more debugging
divagant-martian Sep 5, 2023
39aa67f
actually print the event
divagant-martian Sep 5, 2023
dc8b9a8
validate entries are strictly newer
divagant-martian Sep 6, 2023
01094e5
receive dialer by param in preparation to abstract it away
divagant-martian Sep 6, 2023
d22373e
Revert "receive dialer by param in preparation to abstract it away"
divagant-martian Sep 6, 2023
d8d1512
abstract away the dialer to get rid of the magicEndpoint
divagant-martian Sep 6, 2023
354ab27
propagate changes
divagant-martian Sep 6, 2023
c035640
make trait public to keep the docs as they are
divagant-martian Sep 6, 2023
fe09521
abstract away all the io
divagant-martian Sep 6, 2023
b208ee8
write some invariants and plug them in the main downloader loop
divagant-martian Sep 6, 2023
5ee842a
remove unused piece of code
divagant-martian Sep 6, 2023
3ccbb3f
more invariants
divagant-martian Sep 6, 2023
5787c80
move cancellation vack to the downloader, start smoke test
divagant-martian Sep 6, 2023
f86af83
NO SMOKE HERE!
divagant-martian Sep 7, 2023
20d0626
cleeepeeee
divagant-martian Sep 7, 2023
e359e73
restructure tests
divagant-martian Sep 7, 2023
c78c658
fmt
divagant-martian Sep 7, 2023
1039497
Merge branch 'main' into download-manager
divagant-martian Sep 7, 2023
b04b681
after typo
divagant-martian Sep 7, 2023
ececd4a
add deduplication test
divagant-martian Sep 7, 2023
cba64e2
add cancellation test
divagant-martian Sep 7, 2023
5e80d82
fix invariant check for concurrency limits
divagant-martian Sep 7, 2023
8aa0356
add max concurrent requests test
divagant-martian Sep 7, 2023
5fe0c4f
refactor AvailabilityRegistry to ProviderMap
divagant-martian Sep 8, 2023
0e3907b
remove ProviderMap as a trait
divagant-martian Sep 9, 2023
442b3ac
reduce get code duplication
divagant-martian Sep 9, 2023
aaef3c1
group io get code
divagant-martian Sep 9, 2023
bd7ceb0
better renaming
divagant-martian Sep 9, 2023
1e2c194
add some more docs
divagant-martian Sep 9, 2023
f159a72
cleaner file names
divagant-martian Sep 9, 2023
aec7256
code cleanup
divagant-martian Sep 9, 2023
4f92881
add metrics
divagant-martian Sep 9, 2023
89f6f29
cleanup tests
divagant-martian Sep 9, 2023
ce214d8
make item public to use it in the docs
divagant-martian Sep 9, 2023
a33d8bb
make item public to use it in the docs
divagant-martian Sep 9, 2023
12c63dd
review suggestions
divagant-martian Sep 11, 2023
8ff922d
restore sync tests
divagant-martian Sep 12, 2023
7e704d7
check invariants in debug as well
divagant-martian Sep 12, 2023
aba53f2
misc updates from pr feedback
divagant-martian Sep 12, 2023
4a7e4b2
adjust error management
divagant-martian Sep 12, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 47 additions & 46 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions iroh-bytes/src/protocol/range_spec.rs
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ use smallvec::{smallvec, SmallVec};
///
/// This is a SmallVec so we can avoid allocations for the very common case of a single
/// chunk range.
#[derive(Deserialize, Serialize, PartialEq, Eq, Clone)]
#[derive(Deserialize, Serialize, PartialEq, Eq, Clone, Hash)]
#[repr(transparent)]
pub struct RangeSpec(SmallVec<[u64; 2]>);

Expand Down Expand Up @@ -152,7 +152,7 @@ impl fmt::Debug for RangeSpec {
///
/// This is a smallvec so that we can avoid allocations in the common case of a single child
/// range.
#[derive(Deserialize, Serialize, Debug, PartialEq, Eq, Clone)]
#[derive(Deserialize, Serialize, Debug, PartialEq, Eq, Clone, Hash)]
#[repr(transparent)]
pub struct RangeSpecSeq(SmallVec<[(u64, RangeSpec); 2]>);

Expand Down
2 changes: 1 addition & 1 deletion iroh-gossip/src/net.rs
Original file line number Diff line number Diff line change
Expand Up @@ -374,7 +374,7 @@ impl Actor {
let peer_data = postcard::to_stdvec(&info)?;
self.handle_in_event(InEvent::UpdatePeerData(peer_data.into()), Instant::now()).await?;
}
(peer_id, res) = self.dialer.next() => {
(peer_id, res) = self.dialer.next_conn() => {
match res {
Ok(conn) => {
debug!(?me, peer = ?peer_id, "dial successfull");
Expand Down
Loading
Loading