need separation of private blocks and public blocks

[whyrusleeping]

Before v0.4.0 merges with the pinning updates, we need to be able to have blocks in our blockstore that don't get advertised and sent out to the network. Sending out pin lists would be very bad for privacy and security. This will also enable us to move towards having all ipfs data stored as dags in the blockstore.

One approach to this would be to have two separate blockstore objects with different namespaces into the same underlying datastore. This way, iterating over the keys in our 'public' datastore will not yeild any private data.

[jbenet]

One approach to this would be to have two separate blockstore objects with different namespaces into the same underlying datastore. This way, iterating over the keys in our 'public' datastore will not yeild any private data.

yeah i think this would work

other approaches include:
1. making dags of "allowed" or "blocked" content
2. more granular ACLs where "allowed" or "blocked" can use lists of peers.
3. go fully to a capability system, with read caps

something like (2.) or (3.) is needed in the long run. (2.) will be hard to get right. general ways to do (3.) are well known and work well in practice.

[whyrusleeping]

having dags for allowed or blocked content will get really expensive, as you get more and more content on your system, those dags will become massive, and the cost to percolate the changes up the dag to the root becomes more and more costly with each block you add.

mapping the concept of ACL's to ipfs is gonna be fun. I think we can do it, but it's going to be new territory.

[wking]

On Thu, Jun 18, 2015 at 10:07:45AM -0700, Jeromy Johnson wrote: having dags for allowed or blocked content will get really expensive, as you get more and more content on your system, those dags will become massive, and the cost to percolate the changes up the dag to the root becomes more and more costly with each block you add.

If you have a billion IPFS objects on your system and store them under a ‘public’ DAG (or whatever we call it) with 1028 fanout entries per level, that's only 3 layers deep with an evenly-balanced tree. Rebalancing the tree is going to be expensive, so I think we want to avoid that as much as possible, but bubbling each new public addition up a handful of layers in a fanout tree doesn't seem to painful to me. Maybe @tv42 has a better handle on the scaling, since this tree would probably use a similar implementation to #1225.

[tv42]

What @wking said, the set data type created for pinning should work ok for that. The current settings are items per node=8192 and fanout=256, which would handle 13 billion entries with a 4-level tree.

And it's imitating some of the ideas from fractal trees, so additions should typically touch only the root node.

(Note that the current implementation follows the old pinning API, which assumes in-memory state and full load/store; obviously that won't be fun with 10e9 entries.)