Optimize for memory

[Stebalien]

Currently, we use lots of small maps when storing peer routing information. Instead, we should try to optimize for memory and use arrays. We can temporarily use maps as-needed to update this information but we shouldn't keep them around.

We should be able to halve (or more) peerstore memory usage this way. We'd stop storing multiaddrs twice (once as keys, once as values) and we'd lose the overhead of hash-maps (to avoid collisions hash maps are generally sparse).

[whyrusleeping]

while we're at it, we should start putting the peerstore on disk

[whyrusleeping]

Also, i think there must be a leak or something in any case. mars is currently using 140MB of memory for just the addressbook.

[whyrusleeping]

Also, the way we store information about peers is using keys of map[peerID+key]value. If we instead did map[peerID]map[key]value, we could potentially save a lot on total space it takes to store all the keys. Even going to map[peerID][]KVPair would be fine, given we only have three different keys we really store per peer.

[Stebalien]

It is? From what I can tell, we use map[peerid]map[multiaddr]struct{multiaddr,time}. My plan is to save space by using: map[peerid][]struct{multiaddr,uint64} (no maps, much smaller time value).

[whyrusleeping]

I think we're talking about different spots. youre talking about addresses and i'm talking about where we store agentVersion, protocolVersion and Protocols

[Stebalien]

Ah. My profiles always point to one of the amap[addrs] = expiringAddr{Addr: addr, TTL: exp} lines in addr_manager.go.

[Stebalien]

Fixed.