-
Notifications
You must be signed in to change notification settings - Fork 86
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RAPIDE & IPSL: first proof of concept #148
Conversation
6539fe9
to
7d30d50
Compare
Codecov Report
@@ Coverage Diff @@
## very-fast-downloads ipfs/go-libipfs#148 +/- ##
=======================================================
+ Coverage 18.50% 20.37% +1.86%
=======================================================
Files 96 103 +7
Lines 10337 10807 +470
=======================================================
+ Hits 1913 2202 +289
- Misses 8152 8320 +168
- Partials 272 285 +13
|
Is there a design doc or spec for rapide? The linked issues don't point me to one. |
@iand there is https://www.notion.so/pl-strflt/RAPIDE-3c4fb9e159ae471bb426bb81855cee16 which is a bit outdated (during the implementations I found new tricks which make the implementation much easier, the big arching idea is correct tho). |
Presentation of that work here: https://www.youtube.com/watch?v=JZheMPTxvLc |
78269ef
to
719b5e6
Compare
74a2a52
to
0932d3b
Compare
This still requires lots of work. This include a mostly complete compiler, (minus a few types and literals). More nodes and scopes will be required.
This include a working single threaded structure for RAPIDE. Todo: - Multithreading - dontGoThere map for serverDrivenWorker - Racing
This help avoiding situations where you go race someone else, in a shallow dag, and you get a block next to them in the traversal.
We do not have consensus nor usecase for the language part of IPSL, to avoid long debates for some not yet usefull pieces of code, I've removed it and we will handle merging this part seperately (or never merge it).
In 719b5e6 I changed `download.expand` to release the node lock before returning. This allows other workers to access the just expanded node while we walk up the chain ancestry. This mean that when we relock the node later to add it to our list of tasks, someone else may have downloaded some of the blocks already and thus they are not in the childrens slice and thus we wont add them to our list. This also solve the case where you give a Traversal that is not supported by the underlying protocol, in that case we want to kill the current download and retry deeper, while previously we would hard error on this and kill the worker.
Moved to ipfs#1 |
This support the code for a proof of concept of RAPIDE which concurrently download from multiple gateways.
Todo:
dontGoThere map for serverDrivenWorkerIn memory benchmarks using fake latency andio.Reader
shaping.Include: #33
Fixes ipfs#4 ipfs#2