Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposed DHTv2 Changes #473

Merged
merged 61 commits into from
Apr 6, 2020
Merged

Proposed DHTv2 Changes #473

merged 61 commits into from
Apr 6, 2020

Commits on Apr 3, 2020

  1. upgrade deprecated query events

    aschmahmann authored and Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    57a12ce View commit details
    Browse the repository at this point in the history
  2. first pass at proper kad

    aschmahmann authored and Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    6793dc1 View commit details
    Browse the repository at this point in the history
  3. fix Search/GetValue to be Kad compliant. Default quorum is now 0 whic…

    …h means do not abort the query early
    aschmahmann authored and Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    564764f View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    616b0e1 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    4aee8bb View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    6dac71c View commit details
    Browse the repository at this point in the history
  7. query switches from alpha mode to k mode if no peers closer than one …

    …we have heard about (in a given path) has been found
    aschmahmann authored and Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    85f88e1 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    c7a8d0d View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    cfb4b40 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    19e4d2e View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    11776e0 View commit details
    Browse the repository at this point in the history
  12. options: KValue and AlphaValue global variables no longer used intern…

    …ally. Concurrency option now sets alpha. DisjointPaths option now sets d. Default number of disjoint paths is now bucketSize/2.
    aschmahmann authored and Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    e967aca View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    dd0bbb2 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    4c2cf46 View commit details
    Browse the repository at this point in the history
  15. cleanup unused code

    aschmahmann authored and Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    41f47f7 View commit details
    Browse the repository at this point in the history
  16. Utilize identify events to add peers to the routing table (#472)

    * feat: consume identify events to evaluate routing table addition
    * fix: routing table no longer gets an update just because new messages have arrived or been sent
    * fix: add already connected peers into the routing table before listening to events
    
    Co-authored-by: Raúl Kripalani <raul.kripalani@gmail.com>
    Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
    3 people authored and Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    509c0bc View commit details
    Browse the repository at this point in the history
  17. fix: improve context deadline handling

    1. Continue to best-effort provide, but still return an error when we fail to
    send provider records to the _best_ peers.
    2. Continue returning the best peer's we've found in GetClosestPeers, but also
    return an error to indicate that we didn't find the closest ones.
    
    And fix the hang test.
    Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    1de9b90 View commit details
    Browse the repository at this point in the history
  18. Configuration menu
    Copy the full SHA
    acd3bba View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    052ac59 View commit details
    Browse the repository at this point in the history
  20. Enable switching DHT between client and server modes (#469)

    * created Mode(ModeOpt) option for choosing between auto/client/server modes
    * Auto mode internally switches the DHT between client and server modes based on the EvtLocalReachabilityChanged event emitted on the event bus (e.g. by AutoNAT)
    * routing table management of peers that switch between client and server mode while we are connected to them (i.e. are in auto mode)
    * removed Client(bool) option, becoming a DHT client is specified using Mode(ModeClient) instead
    aschmahmann authored and Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    d4134a4 View commit details
    Browse the repository at this point in the history
  21. feat: move options to main package and make internals private (#486)

    * feat: move options to main package and make internals private
    
    Rationale:
    
    1. This allows us to make private options for testing.
    2. This removes an import for DHT users.
    3. This makes options much easier to discover.
    4. This makes it possible to make the config/options internals private.
    
    We originally put them in a sub-package to avoid poluting the root namespace,
    but that isn't really necessary.
    
    This keeps the old package (for now) to avoid breaking too much.
    Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    30fa086 View commit details
    Browse the repository at this point in the history
  22. Upgrade DHT version (#479)

    * upgraded the protocol id to version 2 (i.e. /kad/2.0.0) and made it so v2 peers running in server mode respond to queries from v1 peers. Note: v2 peers will only send queries using the v2 protocol, will only add v2 peers to their routing tables, and will only tell v1 peers about v2 peers.
    * to run a forked network we now use network specific protocol prefixes instead of manually setting protocol IDs. Use the ProtocolPrefix option instead of the Protocols option.
    * emit errors during initialization if the user misuses the default protocol prefix by setting parameters inconsistent with the default protocol's network specification
    * since the Client option has been deprecated it's been removed from the dht's options. While deprecated it is still available in the dht options package. Setting `Client(false)` now puts the node into ModeAuto.
    aschmahmann authored and Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    07dff1f View commit details
    Browse the repository at this point in the history
  23. RT connectivity changes

    aarshkshah1992 authored and Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    c17965f View commit details
    Browse the repository at this point in the history
  24. Cypress Disjoint Query & Query Termination code cleanup (#489)

    * kpeerset refactoring
    * query code cleanup
    aarshkshah1992 authored and Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    229cacc View commit details
    Browse the repository at this point in the history
  25. change tests

    aarshkshah1992 authored and Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    169395b View commit details
    Browse the repository at this point in the history
  26. document test failure

    aarshkshah1992 authored and Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    8715e2d View commit details
    Browse the repository at this point in the history
  27. Configuration menu
    Copy the full SHA
    5c71aa9 View commit details
    Browse the repository at this point in the history
  28. fix: protect rng with a mutex

    aschmahmann authored and Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    5358a4f View commit details
    Browse the repository at this point in the history
  29. Configuration menu
    Copy the full SHA
    ac9f033 View commit details
    Browse the repository at this point in the history
  30. Configuration menu
    Copy the full SHA
    9b5a0e6 View commit details
    Browse the repository at this point in the history
  31. change peer validation func

    aarshkshah1992 authored and Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    050c4b8 View commit details
    Browse the repository at this point in the history
  32. fix validation

    aarshkshah1992 authored and Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    e1ed825 View commit details
    Browse the repository at this point in the history
  33. queries can take longer

    aarshkshah1992 authored and Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    cbeb73d View commit details
    Browse the repository at this point in the history
  34. Asynchronous lookups (#498)

    * feat(query): fully async implementation of Kademlia lookup. peers returned from the lookup are not guaranteed to be alive (i.e. we're only guaranteed to have dialed the closest beta peers to the target), but given stable and correct routing tables the expectation that most of the peers returned are alive is high.
    * feat(query): add wrapper lookup followup function to followup after the lookup is completed and ensure that the closest k returned peers from a lookup have been queried even for beta < k
    * refactor(query) modified the structure returned from lookups to be a useful subset of the full query state instead of the entire query state
    * feat(options): beta parameter exposed as the Resiliency parameter
    * feat(routing): do not mark the routing table as updated after a FindPeer query
    * feat(routing): FindPeer can return addresses even if not Connected as long as it was either recently connected (CanConnect) or was discovered during the lookup
    * feat(bootstrap): bootstrap logic now uses GetClosestPeers instead of FindPeer
    * refactor(dht): stopFn no longer takes any state
    * fix(test): changed GetClosestPeers test to only assume beta instead of k peers since that is now more appropriate given the query logic changes and that the routing tables in that test are bad, i.e. a ring network with arbitrary peerIDs
    
    Co-authored-by: Petar Maymounkov <petarm@gmail.com>
    Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
    3 people authored and Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    94cf481 View commit details
    Browse the repository at this point in the history
  35. Configuration menu
    Copy the full SHA
    c60ea62 View commit details
    Browse the repository at this point in the history
  36. Remove disjoint queries (#503)

    * remove disjoint queries
    * remove globally queried check
    * removed dht rng field and mutex
    * removed the Disjoint option
    
    Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com>
    2 people authored and Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    94cf3e5 View commit details
    Browse the repository at this point in the history
  37. fix(test): TestFindPeerQuery and TestFindPeerQueryMinimal now refresh…

    … DHT routing tables before running. TestFindPeerQuery no longer runs with the race detector because it uses too many goroutines.
    aschmahmann authored and Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    7c8f404 View commit details
    Browse the repository at this point in the history
  38. Configuration menu
    Copy the full SHA
    fa52ddd View commit details
    Browse the repository at this point in the history
  39. Configurable Peer Filtering (#471)

    Allows specifying peer filter functions on query and on adding peers to the routing table. This patch also includes some reasonable default functions for a public-only and private-only DHT.
    
    Co-authored-by: Will Scott <will@cypherpunk.email>
    Co-authored-by: Steven Allen <steven@stebalien.com>
    3 people committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    0543571 View commit details
    Browse the repository at this point in the history
  40. bump the peerstore version

    aarshkshah1992 authored and Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    29bc71a View commit details
    Browse the repository at this point in the history
  41. feat: reduce request read timeout (#527)

    This should stop us from waiting on unresponsive peers. This only kicks in when
    we've already _sent_ a request so this:
    
    * Shouldn't be blocked on other requests we're planning on making.
    * Shouldn't be blocked on dialing.
    Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    fad6c8a View commit details
    Browse the repository at this point in the history
  42. Emit events during async query. (#504)

    * Add Kademlia async lookup implementation (replaces prior implementation)
    * Add lookup events that allow full lookup reconstruction from logs
    * Introduce lookup stopping condition, based on parameter "beta"
    
    Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com>
    2 people authored and Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    5a41195 View commit details
    Browse the repository at this point in the history
  43. Configuration menu
    Copy the full SHA
    39990c0 View commit details
    Browse the repository at this point in the history
  44. feat: return target peer if we know about them (#511)

    Previously, we'd only return the target peer if we were connected to them.
    However, this makes it difficult to impossible to find peers that are mostly
    disconnected from the network.
    
    This change also changes `p` to `from` in several places as `p` is _very_
    confusing. We should probably switch away from using `p` everywhere.
    Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    d0770d2 View commit details
    Browse the repository at this point in the history
  45. New RT management policy (#520)

    * new RT management policy
    aarshkshah1992 authored and Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    8fa0438 View commit details
    Browse the repository at this point in the history
  46. Configuration menu
    Copy the full SHA
    d715495 View commit details
    Browse the repository at this point in the history
  47. Periodic Pinging and active fixLowPeers (#533)

    * fix Low Peers and Cleanup
    aarshkshah1992 authored and Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    2539114 View commit details
    Browse the repository at this point in the history
  48. Record valuable peers will only take into account latency of seed pee…

    …rs (#536)
    
    * record valuable peers algorithm only takes into account latency of seed peers
    aschmahmann authored and Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    6a36698 View commit details
    Browse the repository at this point in the history
  49. chore: update deps

    Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    485c60a View commit details
    Browse the repository at this point in the history
  50. refreshing the routing table after the dht context is cancelled retur…

    …ns an error instead of hanging
    aschmahmann authored and Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    a4eaa70 View commit details
    Browse the repository at this point in the history
  51. chore: fix lints

    Except deprecated warn calls
    Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    5aad954 View commit details
    Browse the repository at this point in the history
  52. feat: better logging

    * Remove deprecated events.
    * Fix warning -> warn.
    * Use structured logging (some places).
    Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    445472b View commit details
    Browse the repository at this point in the history
  53. Configuration menu
    Copy the full SHA
    dfea791 View commit details
    Browse the repository at this point in the history
  54. fix(test): don't panic on read/write error

    The querying node _is_ allowed to reset and go away.
    
    fixes #431
    Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    d520a1e View commit details
    Browse the repository at this point in the history
  55. chore(ci): remove travis

    Stebalien committed Apr 3, 2020
    Configuration menu
    Copy the full SHA
    deee8b7 View commit details
    Browse the repository at this point in the history

Commits on Apr 5, 2020

  1. Configuration menu
    Copy the full SHA
    5e318d3 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    6f1df26 View commit details
    Browse the repository at this point in the history

Commits on Apr 6, 2020

  1. feat: add a v1 compatibility mode

    When this option is passed, the DHT node will listen on and query _only_ using
    the old DHT protocol. Importantly, the node won't even pretend to be a new DHT
    node because it's routing table includes V1 peers.
    Stebalien committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    9bab969 View commit details
    Browse the repository at this point in the history
  2. chore: remove unused testProtocols

    We no longer need this now that we explicitly support v1 compatibility mode.
    Stebalien committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    8af12dd View commit details
    Browse the repository at this point in the history
  3. chore: update kbucket

    Stebalien committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    7035c59 View commit details
    Browse the repository at this point in the history
  4. test: fix TestFindPeerQuery and TestFindPeerQueryMinimal to not try a…

    …nd refresh routing tables for orphaned nodes
    aschmahmann authored and Stebalien committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    a7fd35f View commit details
    Browse the repository at this point in the history