calculate network size dynamically

[lukemarsden]

ETA: 2023Q1

new performance improvements require nodes to know a lower bound on the size of the network, currently we hard-code this in #511 but we need to calculate this dynamically. Until we fix this, big networks won't be fast again (since I just changed that number from a hard-coded 250 to a hard-coded 3)

[lukemarsden]

262477d

[lukemarsden]

we can probably start by just keeping track of how many distinct node ids we've heard from in the last... say, 10 minutes? and use that in our calculation

[lukemarsden]

ah but then we should probably also make nodes heartbeat on the network every minute or so so that quiet nodes don't disappear from the calculation

[binocarlos]

the heartbeat could contain info about "here is the capacity that I offer the network" - then we can:

• know how many nodes there are
• know how many resources there are across the whole network
• know how many resources a single node offers
  • this will help requester nodes to accept certain bids based on each nodes reported capacity

[aronchick]

fascinating problem - how does K8s do this? Do kubelets heartbeat back to apiserver?

[lukemarsden]

Via etcd I believe yes

[wdbaruni]

This should not be a problem once we switch from GossipSub to direct calls for job negotiation

[wdbaruni]

I am closing this task as we shouldn't solve it since we will drop adding artificial latencies before bidding on jobs