Feature Request: Mark Dataset As Private

[nathantchan]

Folks may want to benefit from both the distributed web and version controlling of datasets as independent features. They may not want to make some of their data public via the distributed web by running connect. It would be great if there was an easy way in a dataset's metadata to make explicit that you don't want it to be shared.

[b5]

While you're not the first to ask for private data support, you are the first to file an issue on the topic 😄. To me this feature falls nicely in line with npm's private package flag, and I totally agree it'd be great to replicate this pattern in qri datasets.

There's one hangup. When peers run qri add, that data is ingested into the underlying IPFS node, which makes it available on IPFS. This means it's possible for someone to check your public peer ID, and request the list of hashes you have pinned, one of which would be this private dataset if you were online. So while it'd be relatively trivial to add a private flag to our dataset definition that prevents peers from disclosing the existence of a dataset, we want to make sure that data isn't sharable in any way. I'm planning on working with the IPFS team to sort through how our work will intersect with any plans for private hashes, and we'll be shipping some features on the issue while that discussion progresses (encrypt-before-addition, storing private data outside of IPFS).

So with that, I think we can ship this feature in a few phases:

Add a non-working private flag with info

In this case we'll add private flag to the CLI & API that returns an error if set to true, informing users that private data is currently unsupported, and refer them to a github issue to track progress. This is an important first step we can ship now to help clear up expectations around private data.

Publish a private data roadmap RFC

There are a few things I'd like to have roadmapped before we start shipping any form of private data features. This roadmap should plot the course for access control, private networks and security. All three of these when finished should add up to a robust framework for private data on qri. We'll have a period for public feedback, for which I'd love your input!

Implement the roadmap itself in phases

From there we'll set to work implementing the roadmap itself, starting with the features that affect the largest number of use cases.

Complete a security audit

We can't call data private until someone who isn't us has looked at our implementation of private data and given us a degree of confidence that we haven't done anything stupid. I plan to release private data features before we've completed this with a big warning until we've properly completed at least one of these audits. This last bit is going to be painful, slow, and incredibly important.

It'll be at least 6 weeks (possibly much longer, no promises 😉) before we see the first signs of this feature realized, but we absolutely plan on supporting private data, and need input on the subject, so thanks for bringing this up @nathantchan!

cc @rgardaphe, who's coordinating a few other's private data interests.