New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Private sites #62

Open
HelloZeroNet opened this Issue Mar 16, 2015 · 27 comments

Comments

Projects
None yet
5 participants
@HelloZeroNet
Owner

HelloZeroNet commented Mar 16, 2015

Sites protected by password or public-key based auth.

@OliverCole

This comment has been minimized.

Contributor

OliverCole commented Mar 2, 2016

I noticed you haven't labelled this one with help wanted. Is that intentional? Do you have a particular design in mind you could share?

This would be something I'd love to implement!

@HelloZeroNet

This comment has been minimized.

Owner

HelloZeroNet commented Mar 6, 2016

@OliverCole AES and ECIES encrypt/decrypt functions added some months ago: #216
It could allow to encrypt every content on the site and decrypt after it downloaded on the client side.

It's not ideal for every application case, there is many ways to implement private sites it's all depends on what you want to do? Filter IP-s who is connecting to your site? Add password protection? Only encrypt part of the sites?

@OliverCole

This comment has been minimized.

Contributor

OliverCole commented Mar 6, 2016

@HelloZeroNet Yeah, I think we have similar thoughts on this.

To me you want to have the public site unencrypted, and then an encrypted 'subdirectory', or even more than one. This means the owner can have public content, and how to contact them perhaps, and then the private stuff all under one key/bit domain. They can always choose to keep the public site blank if they wish.

We should think about each private 'subsite' as a site itself though - with it's own content.json etc. I feel like it should probably be rolled up in a zip file to avoid any visibility of the private site structure, but then that might make publishing difficult?

Some other thoughts:

  • You can support various access methods:
    • Simple 'password', entered into ZeroNet managed UI over a site - stretched to the shared decryption key
    • Certificate access. This would require the owner encrypting the decryption key to each cert, and publishing the lot, but it's super easy for the users, they just know that a given certificate grants access. This does also mean you can rotate the underlying decryption key - on every site publish if required!
    • Bitcoin integration? Further in the future, what about something that would publish copies of the decryption key, encrypted to the public key of Bitcoin addresses which had sent it an amount of Bitcoin.
  • You could manage the whole workflow quite neatly - users could write an access request, which would be encrypted to the site owner and published. The site owner could have an interface for reviewing them and granting access where needed.
  • I'm not sure IP filtering is worth considering because it's not guaranteed for a user, and some people are deliberately using ZeroNet through Tor.

It's definitely not ideal for all use cases - and in particular, any user with access can then distribute (or reveal) that access to other users, regardless of whatever UI we put over the top. But I think it would be pretty good within the limitations of ZeroNet.

@alxbob

This comment has been minimized.

alxbob commented Mar 15, 2016

Maybe using something like this https://github.com/pyca/pynacl used in openbazzar

@HelloZeroNet

This comment has been minimized.

Owner

HelloZeroNet commented Mar 15, 2016

We already using openssl, pynacl just an alternative to it and it does not support bitcoin cryptography that we using most of the time

@OliverCole

This comment has been minimized.

Contributor

OliverCole commented Mar 25, 2016

So @HelloZeroNet, I want to have a crack at this - maybe just starting with the encryption/publishing part, for the 'simple password' case. Are you happy to merge if we come up with something good?

My first thought is on the metadata - we could add this in the files list in content.json, but that's not strictly accurate. Alternatively we could call it an include.

But I think a third list might be a good idea, eg:

{
 "address": "1evJheeFpVQkHaZdAzEMj75w5T14ogZWT",
 "description": "My example site",
 "files": {
  "index.html": {
   "sha512": "dc5af04d2cde806f4bab9a0dd6b09d367c9f2dd51dd5f22a36c2dc56aa6e925e",
   "size": 428
  }
 },
 "private": {
  "members": {
   "description": "Private members area - contact johnsmith on ZeroMail for access",
  }
 },
...

Signing and publishing would then require the encryption key (not ZIP password! 😝), stretch it, and encrypt members.zip to members.zip.zeronet - the encrypted file gets published and the cleartext remains on disk.

members.zip would contain content.json at the top level, which would behave like any other subdirectory content.json. And members.zip.zeronet would be eligible for distribution by users, along with the rest of the site.

Later on, when we get to retrieving, a request for 1evJheeFpVQkHaZdAzEMj75w5T14ogZWT/members/ would trigger ZeroNet to request the key for members.zip.zeronet, decrypt it on disk and then proceed as normal.

@HelloZeroNet

This comment has been minimized.

Owner

HelloZeroNet commented Mar 25, 2016

Not sure if storing everything twice is a good idea, probably a password/publickey based peer authorization is easier and would allow to remove users.

@OliverCole

This comment has been minimized.

Contributor

OliverCole commented Mar 25, 2016

How do you mean twice? The only thing that would actually be shared out would be the encrypted zip.

Are you proposing the private bit is unencrypted, but only shared to people who should have access? Could you outline how that would work?

@HelloZeroNet

This comment has been minimized.

Owner

HelloZeroNet commented Mar 25, 2016

To be able request any file from peers you need to authenticate yourself using publickey algorithm.

Other possibility: when requesting any.jpg using http and it's not exists, then it's looks for any.jpg.encrypted and try to decrypt it using they AES keys you have associated with the site/directory.

This way the sign/publish method would remain the same, also allows multi-user sites and only requires some smaller modifications in the UiServer/SiteStorage (for sql imports) and new API functions to add AES keys.

@HelloZeroNet

This comment has been minimized.

Owner

HelloZeroNet commented Mar 25, 2016

I think there is two possibility:

Per file encryption

Anyone able to receive files/updates, but the files are encrypted

Pros:

  • Does not relies on connection security
  • Allows easy per-directory encryption
  • Allow anyone to host files without knowing it's content (eg. paid hosting)

Cons:

  • Performance: has to decrypt the files on the fly or build a cache
  • Not possible to remove users from the site
  • You can spy on site activity
  • (Future) Patch command can be problematic

Connection security

You need to authenticate with other users connection before receive any updates or files.
Pros:

  • Not possible to spy on site activity
  • May be possible to create new password and remove users
  • Files stored the same way as any other site

Cons:

  • Relies on connection security (SSL/onion): MITM can be a problem
  • Per directory encryption is harder, but may be possible
@OliverCole

This comment has been minimized.

Contributor

OliverCole commented Mar 26, 2016

You can spy on site activity

Are you talking about metadata, like file names/sizes etc? That's partly what the zip file is for - just to bundle it up and avoid that analysis.

(Future) Patch command can be problematic

I haven't read anything about this - what is it?

You can actually still achieve adding/removing users with per-file encryption, by picking a random symmetric key, encrypting the files (or zip file), and then encrypting a copy of that key to each user, exactly the same way PGP does it. If you want to add a user, simply publish an additional encrypted copy. Removing a user means picking a new random key and encrypting that for the remaining users.

I think per-file encryption is the best, but tell me about the patch command?

@HelloZeroNet

This comment has been minimized.

Owner

HelloZeroNet commented Mar 26, 2016

the problem is the zip file is not suitable for multi-user sites and per-user encryption is not really works for many users (100+)

patch command: to greatly reduce bw usage when one of the files modified instead of re-transfer the whole file only send the changed lines (diff)

@OliverCole

This comment has been minimized.

Contributor

OliverCole commented Mar 26, 2016

multi-user sites

You mean public zeronet proxies? True... but are you thinking that if it was per-file, it would be decrypted in the browser? Otherwise you still have to trust the server with the key.

per-user encryption is not really works for many users (100+)

Surely that's only (number of users * encrypted 256 bit key)? That's 32 users/kilobyte, minus some for overhead - doesn't seem insane to me.

Patch command is interesting.

Something else to consider is whether it is desirable for users without access (in the per-file model) to already be distributing the encrypted files, without access. Think about the scenario where Assange posted those diplomatic cables encrypted in bulk, as an insurance policy... you would want them to be replicated by interested parties, even if nobody but the author had a key.

I would say it would be good if ZeroNet supported that use case. In the connection security model, it wouldn't be possible because nobody would have access.

@HelloZeroNet

This comment has been minimized.

Owner

HelloZeroNet commented Mar 27, 2016

Multi-user sites: interactive zeronet sites where every user has his/her own files. (ZeroTalk, ZeroBlog, ZeroMail, etc.)

If the files are encrypted and you want to remove an user, then you have to re-encrypt every file and everyone has to re-download all of them.

If you want to make sure of encryption you will still able to encrypt the data using AES+ECIES functions (like ZeroMail)

@OliverCole

This comment has been minimized.

Contributor

OliverCole commented Mar 27, 2016

Well, there is a way around having to re-encrypt all the files - version and increment the keys.

  1. Files A and B (in the main part of the site) are encrypted with K1, and U1 knows K1 because the site owner encrypted a copy of it with KU1.
  2. U2 publishes file C, and encrypts it with K1.
  3. Site owner 'deletes' U1. Site owner generates a new shared key K2, and writes a new table, with a copy of K2 encrypted to KU2, KU3 etc.
  4. From this point, K1 is considered 'compromised', and new and updated files will only use K2.
  5. Site owner makes a change to A, encrypts it with K2.
  6. U2 publishes new file D, encrypts it with K2.

So U1 can still access the files as they were when they had access (until they're republished anyway), BUT that's OK because they always had access to those files, and could have taken a copy at any time.

Patch command

Could we make this work by simply encrypting the patches?

  • In the basic scenario, you simply send the patch encrypted with K1. Each node decrypts the file, applies the patch and re-encrypts.
  • In the advanced case where you've deleted a user, and are updating a file for the first time since, you encrypt the patch with K2, and set a flag to say that each node will decrypt the file with K1, apply the patch, and re-encrypt it with K2. Every node has to repeat the work, but that's not so bad.
@HelloZeroNet

This comment has been minimized.

Owner

HelloZeroNet commented Mar 27, 2016

New key release could work, but then you have to also keep the old keys in order to able to decrypt the old contents. ECIES takes more space, than symmetric encryption: 225 Bytes base64 encoded, so 10000 user = 2.2 Mbyte new data added to site when someone is removed.

It could be possible to encrypt the patch with some overhead: decrypt patch -> decrypt the whole file -> apply patch -> encrypt the file -> check sha512

It would be nice to check how other projects do this (syncthing, bittorrent sync, Tahoe-LAFS, etc.)

@OliverCole

This comment has been minimized.

Contributor

OliverCole commented Mar 28, 2016

Actually, thinking about the patch command... you said the zip file wouldn't work because of the patch command: the patch could simply include a path into the zip file. It would be interesting to do some experiments and see if you could deterministically produce the same patched encrypted zip file on multiple hosts/OSs. It should be possible though.

But anyway, there are a lot of questions and complexity here, and we'll never get the right answer straight away. Why don't we implement the simplest possible idea: single shared AES per-file encryption (ie, image.jpg.encrypted), get it working with the patch command and new key management GUI, and get it out there to see how people use it? Or indeed if anyone does actually bother using it at all!

That way we avoid the complexity of adding/removing individual users, managing access etc, and we don't spend a bunch of time building something people won't use.

@HelloZeroNet

This comment has been minimized.

Owner

HelloZeroNet commented Mar 28, 2016

The per-file encryption is not compatible with patch command because it's not a good idea to re-use the IV. if we add a new feature we have to support it forever, so it's need to be as flexible as possible.

@OliverCole

This comment has been minimized.

Contributor

OliverCole commented Mar 29, 2016

By reuse, you mean it's not a good idea to encrypt the first version of the file with {K1,IV1}, encrypt the patch with {K1,IV1}, decrypt the file, make the change, and then re-encrypt it with {K1,IV1} again? You're right, that would be bad.

However, we can send IV in the clear without risk. So we would need to use a fresh IV for the patch encryption, and the patch would need to contain another fresh IV to use after the patch has been applied. We might want to include checksums for both plaintext and ciphertext inside the patch.

However, IV presents a bigger question for multi-user sites - we would need to use fresh IVs for user publications, and publish those along with the ciphertext. The sqlite DB only exists locally, and never leaves the machine, right? Or do we need to think about that too?

In fact, if we don't use zip files, we have to publish IVs per file too. On a site with a lot of files that could eat a lot of entropy from the pool.

@OliverCole

This comment has been minimized.

Contributor

OliverCole commented Apr 3, 2016

It's probably worth trying to put this all together in a wiki page that shows all the algos, keys and messages... any thoughts before I get started?

@HelloZeroNet

This comment has been minimized.

Owner

HelloZeroNet commented Apr 4, 2016

I think it's fine to put it here

@mirrorwish

This comment has been minimized.

mirrorwish commented Apr 8, 2016

In fact, if we don't use zip files, we have to publish IVs per file too. On a site with a lot of files that could eat a lot of entropy from the pool.

The IVs can be put inside the file along with the ciphertext and the MAC. Patches could send a single random value and the IVs will be derived using HKDF, but it shouldn't really be a problem either way.

@anoadragon453

This comment has been minimized.

Contributor

anoadragon453 commented Jun 30, 2017

Glad to see there's some talk on this. I'd like to throw in the idea of enabling the ability to create multiple passwords/secret addresses (a la Tor's HidServAuth config command [spec]) to give out to multiple trusted users.

That way if a password or the address were ever compromised, the site owner could simply disable the compromised user's address/password, while everyone else's would still work.

@OliverCole

This comment has been minimized.

Contributor

OliverCole commented Jun 30, 2017

@anoadragon453 Absolutely, I started outlining some ideas at https://github.com/OliverCole/ZeroNet/wiki/Private-sites-on-ZeroNet, but haven't had time to work on it further. Let me know if you want to talk over what I wrote - it wasn't finished.

The problem is it gets complex fast - when you remove a user, do you want to re-encrypt the site content to immediately deny them access to the data? Or just encrypt changes from that point? How do you make all that work with the delta updates?

But I think there is huge value in supporting a simple use case - someone publishes some leaked documents with a single symmetric key, which is revealed to media/the public at a later date. I would love to see Zeronet do that!

@anoadragon453

This comment has been minimized.

Contributor

anoadragon453 commented Jun 6, 2018

I think this could open a lot of really novel use cases. I also think the authenticated method is the best solution. It requires much less invasive changes to ZeroNet, and because the "someone publishes some leaked documents with a single symmetric key, which is revealed to media/the public at a later date" bit can already be done with ZeroNet really. Just encrypt the file locally and share the file on a zite.

The password-based authentication method can be added, though imo pubkey-based first sounds like the best solution, as I imagine that'll be the more widely used one.

The per-file encryption feature is nice is theory, but quite computationally expensive and not scalable. Doing it on an authenticated request basis is much cheaper, and per-directory authentication with authentication is trivial (like web servers do so today).

There is currently a 0.169BTC bounty (or $1261.37) open for this, and adding private zites would start to allow for paid content, and giving ZeroNet sites an avenue for legitimately distributing copyrighted works over an entirely decentralized system.

@OliverCole

This comment has been minimized.

Contributor

OliverCole commented Jun 6, 2018

I likely don’t have time to work on this any more, so anyone should feel free to have a go.
All I would suggest is making sure to support various ‘versions’ of private sites - so that if we want to support more advanced use cases/cryptographic structures later we still can.

I didn’t know there were bounties on ZeroNet issues though - is there a link?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment