End-to-end encryption (E2E) between clients (Android app, CLI, web app)

[binwiederhier]

It should be possible to e2e encrypt messages, like this:

NTFY_PASSWORD=... ntfy publish --encrypt mytopic my encrypted message

Or this

echo my encrypted message | <something> | curl -T- -H "Encrypted: yes" ntfy.sh/mytopic

The message should look like this:

{
  "id": "..",
  "type": "message",
  "encryption": "aes-128-gcm",
  "message": "ZnNkZmRzZnNkZmZzZGtqZmRzamZsc2RmCg=="
}

It is important to me that the solution is widely supported and can be easily implemented in all languages. Ideally something like gpg or age, but I doubt such a format exists.

[binwiederhier]

I looked at this a little today, and sadly had to rule out gpg, because the library for Go is deprecated, and I don't want to build on deprecated technology. I also looked at age a while ago and even opened an issue here (FiloSottile/age#404) to ask for support in other languages; and sadly there is not enough support.

I also played with openssl to see if I can encrypt and protect the integrity (~ AEAD-style), but I wasn't successful.

I will try and play some more, but I believe I'll have to implement my own format. Something similar to what I have done here (https://syncany.readthedocs.io/en/latest/security.html#encrypting-new-files), though much simpler hopefully.

[binwiederhier]

Here's the basic message structure: #354. This implements the exact same thing that Pushbullet does (https://docs.pushbullet.com/#encryption) and is even compatible with it. This was super easy. The trickier part is figuring out the UX of this. I'll have to think about that.

[binwiederhier]

This can encrypt a message with a key, but a key is not a password. The password we'll still have to derive using pbkdf2.

The question is now:

• Does each topic have its own password?
• Or is it somehow tied to the user? If so, pub-sub doesn't really work anymore, because the ciphertext is now different for each user.
• Where do we enter a topic password? What happens if it's wrong?

Relatively easy would be:

• Each subscription has a password.
• The password is stored/entered in the subscription settings and stored locally
• If an encrypted message comes in, we derive the key using the password and use it to decrypt the message. If it succeeds, it's displayed. If not, we show (encrypted message)
• If an unencrypted message comes in, we do what we do now.

[binwiederhier]

Even though I have not updated this ticket, I have been working on it in the #354 branch. I've explored JWE as a solution for the crypto format, and implemented PoCs in PHP and Python and Go.

After realizing that that won't work for attachments, I have switched gears after a discussion with a co-worker and with @wunter8.

Proposal

General flow

• Each subscription has a password.
• The password is stored/entered in the subscription settings and stored locally
• If an encrypted message comes in, we derive the key using the password and use it to decrypt the message. If it succeeds, it's displayed. If not, we show (encrypted message)
• If an unencrypted message comes in, we do what we do now.

Key derivation

• PBKDF2 with 50k rounds and sha256(topicURL) as a salt (see WIP: Crypto stuff #354 for example impl)
• The resulting key is used below

Encryption file format

• Message or attachment are encrypted with AES-256-GCM using the key above
• The format is "ntfy2586v1" || tag (128 bits) || iv (96 bits) || ciphertext [aes-256-gcm]
• (This is not implemented yet)

Publishing (without attachment)

PUT /mytopic HTTP/1.1
Encrypted: yes

<encrypted binary data, ntfy format>

Publishing (with attachment)

PUT /mytopic HTTP/1.1
Encrypted: yes
Content-Type: multipart/form-data; boundary=--ntfy-boundaryXyASDsdA

----ntfy-boundaryXyASDsdA
Content-Disposition: form-data; name="message"
Content-Length: 1234

<encrypted binary data, ntfy format>

----ntfy-boundaryXyASDsdA
Content-Disposition: form-data; name="attachment"
Content-Length: 42343242

<encrypted binary data, ntfy format>

----ntfy-boundaryXyASDsdA--

[goalieca]

Password:

• minimum 6 characters, max 255.
• simple byte-string (implementations need to be consistent though, presumably utf-8 source, no null termination)
• can also be thought of as a generic secret (eg: could use yubikey to generate a strong key)

KDF:

• Should specify the URI is normalized. Non-normalized forms will derive different keys.
• I'm still thinking about specifying the number of rounds as a parameter? crypto agility would say yes. but then that would need to be stored alongside the password or in the jwe header along with kid. That's messy. That's the fun with 'dir' mode.

[goalieca]

It might also be worth adding 'kid' to the JWEs so that can give users a chance to change or rotate their passwords. Changing the passwords has some choices because of historically encrypted messages. Since the user holds the password it would be up to them to have to re-encrypt every message.

Using a key-encryption-key (KEK) to encrypt the JWEs would provide better security. The KEK would be wrapped by the password and the client would store the wrapped KEK somewhere. It could be password wrapped, it could be key-vaulted, who knows. How to transport that KEK between endpoints? iOS has cloud-sync as an example. Or if it was wrapped by the password (pbkdf2) then could nfty itself do it? Rotation the password on the KEK is easier as there's just one. Still hard to rotate the KEK though.

I'll think some more on ergonomics of this.

[binwiederhier]

I am still working on this. It's going slow (mostly due to personal commitments), but I have still managed to come up with a design that I think I like. So here it goes:

Proposal 7/13

General flow

• Each subscription has a password.
• The password is entered in the subscription settings
• We derive the key and locally store the key with the subscription
• If an encrypted message comes in, decrypt the message. If it succeeds, it's displayed. If not, we show (encrypted message)
• If an unencrypted message comes in, we do what we do now.

Key derivation

• PBKDF2 with 50,000 rounds and sha256(topic) as a salt (e.g. sha256(mytopic))
• The resulting key is used below

Encryption file format

• Message (and optionally attachment) are encrypyted as JWE
• As of now, only AES-256-GCM in "dir" mode ({"alg":"dir","enc":A256GCM"}) is supported

Use cases

1. Unencrypted message (headers)

Request (HTTP)

POST /sometopic HTTP/1.1
Title: some title

some message

Request (via curl)

curl -d "some message" -H "Title: some title" ntfy.sh/sometopic

Response

{
  "id":"eWa5epsDHkQ4",
  "time":1657759261,
  "event":"message",
  "topic":"sometopic",
  "title":"some title",
  "message":"some message"
}

2. Unencrypted message (JSON)

Request (HTTP)

POST / HTTP/1.1

{
  "topic":"sometopic",
  "title":"some title",
  "message":"some message"
}

Request (via curl)

curl ntfy.sh -d '{
  "topic":"sometopic",
  "title":"some title",
  "message":"some message"
}'

Response

{
  "id":"eWa5epsDHkQ4",
  "time":1657759261,
  "event":"message",
  "topic":"sometopic",
  "title":"some title",
  "message":"some message"
}

3. Unencrypted message with attachment (headers + attachment)

Request (HTTP)

POST /sometopic HTTP/1.1
Title: some title
Message: some message

<attachment data>

Request (via curl)

curl -T attachment.jpg -H "Title: some title" -H "Message: some message" ntfy.sh/sometopic

Response

{
  "id":"eWa5epsDHkQ4",
  "time":1657759261,
  "event":"message",
  "topic":"sometopic",
  "title":"some title",
  "message":"some message",
  "attachment": {
    "name": "attachment.jpg",
    "type": "image/jpeg",
    "size": 715814,
    "expires": 1657775583,
    "url": "https://ntfy.sh/file/tmhElNL0MiKM.jpg"
  }
}

4. Unencrypted message with attachment (multipart: JSON + attachment) [** new]

Request (HTTP)

POST / HTTP/1.1
Content-Type: multipart/form-data; boundary=--ntfy-boundaryXyASDsdA

----ntfy-boundaryXyASDsdA
Content-Disposition: form-data; name="message"

{
  "topic":"sometopic",
  "title":"some title",
  "message":"some message"
}

----ntfy-boundaryXyASDsdA
Content-Disposition: form-data; name="attachment"

<attachment data>

----ntfy-boundaryXyASDsdA--

Request (via curl)

curl ntfy.sh \
  -F attachment=@attachment.jpg \
  -F message='{
    "topic":"sometopic",
    "title":"some title",
    "message":"some message"
  }'

Response

{
  "id":"eWa5epsDHkQ4",
  "time":1657759261,
  "event":"message",
  "topic":"sometopic",
  "title":"some title",
  "message":"some message",
  "attachment": {
    "name": "attachment.jpg",
    "type": "image/jpeg",
    "size": 715814,
    "expires": 1657775583,
    "url": "https://ntfy.sh/file/tmhElNL0MiKM.jpg"
  }
}

5. Encrypted message (JWE-encrypted JSON) [** new]

Request (HTTP)

POST /sometopic HTTP/1.1
Encoding: jwe

eyJhbGciOiJka.............-nH7ya1VQ_Y6ebT1w.2eyLaTUfc_rpKaZr4-5I1Q

Request (via curl)

curl ntfy.sh/sometopic \
  -H "Encoding: jwe" \
  -d "eyJhbGciOiJka.............-nH7ya1VQ_Y6ebT1w.2eyLaTUfc_rpKaZr4-5I1Q"

Response

{
  "id":"eWa5epsDHkQ4",
  "time":1657759261,
  "event":"message",
  "topic":"sometopic",
  "message":"eyJhbGciOiJka.............-nH7ya1VQ_Y6ebT1w.2eyLaTUfc_rpKaZr4-5I1Q",
  "encoding":"jwe"
}

6. Encrypted with attachment (multipart: JWE-encrypted JSON + JWE-encrypted attachment) [** new]

Request (HTTP)

POST /sometopic HTTP/1.1
Encoding: jwe
Content-Type: multipart/form-data; boundary=--ntfy-boundaryXyASDsdA

----ntfy-boundaryXyASDsdA
Content-Disposition: form-data; name="message"

eyJhbGciOiJka.............-nH7ya1VQ_Y6ebT1w.2eyLaTUfc_rpKaZr4-5I1Q

----ntfy-boundaryXyASDsdA
Content-Disposition: form-data; name="attachment"

eyJhbGciOiJk......(this could be 15 MB of data) .........Rzs4Y5QLE2XD2_aw_SQ.y2hadrN5b2LEw7_PJHhbcA

----ntfy-boundaryXyASDsdA--

Request (via curl)

curl ntfy.sh/sometopic \
  -H "Encoding: jwe" \
  -F attachment=@encrypted-attachment.bin \
  -F message=@encrypted-message.bin'

Response

{
  "id":"eWa5epsDHkQ4",
  "time":1657759261,
  "event":"message",
  "topic":"sometopic",
  "message":"eyJhbGciOiJka.............-nH7ya1VQ_Y6ebT1w.2eyLaTUfc_rpKaZr4-5I1Q",
  "encoding":"jwe",
  "attachment": {
    "name": "attachment.bin",
    "type": "application/octet-stream",
    "size": 715814,
    "expires": 1657775583,
    "url": "https://ntfy.sh/file/tmhElNL0MiKM.bin",
    "encoding":"jwe",  // <<<<<<<<<< Do we need this? 
  }
}

Issues/questions

1. JWE has a 30% overhead due to base64; that's okay for messages, but a little sad for attachments
2. The salt for PBKDF2 is the topic URL. In some cases, endpoints are different and clients will derive an "incorrect key". Any ideas for a better salt? We'll use sha256(topic) instead
3. For use case 5+6, the topic+encoding has to be passed as header. That ok, right?

[goalieca]

1. base64 does appear to compress well with gzip (https://blog.virtual-void.net/base64-vs-gzip/)

[steelman]

NTFY_PASSWORD=... ntfy publish --encrypt mytopic my encrypted message

Friendly reminder: environment variables are only little less bad means to pass secrets than command line. Otherwise +1 (or more).

[aksdb]

I want to throw in two considerations regarding the decision against "age" and "openpgp":

1. The Go OpenPGP implementation is still maintained. The one that was in the golang.org/x tree is no longer maintained, but they even referenced the existing fork from ProtonMail as a successor for people who continue using OpenPGP. We use it in production, for example. (Just as ProtonMail does, I would assume.)
2. While age does not have existing implementations in a lot of languages, rolling your own scheme doesn't either. If you start implementing a scheme, you might as well use one with a spec. The advantage over a custom / own scheme is obviously that it is already more battle tested and you have reference implementations to test against.

[binwiederhier]

@aksdb Thanks for the comments.

Re 1: Thanks for the correction. I was not aware that there was another active implementation.

Re 2: The scheme that I picked is not a roll-your-own. It's a subset of JWE. I picked very small subset, so that I could implement it in many different languages easily. I already did it in Python, PHP, Go, and (partially) JS. It is understandable that you think it's roll-your-own, because this ticket discussion is long and old, but this is the one I picked: #69 (comment) -- which is JWE with AES-256-GCM in "dir" mode ({"alg":"dir","enc":A256GCM"}).

Re 2 (age): I did approach the age devs in the GitHub discussions asking for different implementations, and while there is interest, it doesn't seem like it's been done or actively worked on/maintained. Most of the chosen scheme I could implement in 10 lines of code in most languages. It's just AES-256-GCM and a bunch of base64-ing. Simple and I've done that many many times. I'm not new to this :)

[aksdb]

Sounds good, thanks for the clarification (and all the effort you put into this project).

[blewsky]

Awesome project @binwiederhier! Great work.
Password protection would be a really great feature that would make the tool a lot more powerful.

A small question (I have no crypto background): What would happen if two people used (e.g. by accident) the same topic with a different password? Would that be possible? I guess they wouldn't receive the other person's notification and only their own (because they can only decipher their own message)?! Would they get an error message of an attempted notification/wrong password?

[Fysac]

Regarding key derivation: since this is a new design and you have a choice, it doesn't make much sense to use PBKDF2. Instead, a memory-hard hash function like Argon2 or scrypt is far preferable.

If you ultimately stick with PBKDF2, the proposed number of iterations (50,000) is too low to defend against brute-force attacks on modern hardware. 1Password uses 650,000 iterations now, and OWASP recommends at least 600,000 iterations of PBKDF2-HMAC-SHA256: https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html#pbkdf2

[binwiederhier]

Regarding key derivation: since this is a new design and you have a choice, it doesn't make much sense to use PBKDF2. Instead, a memory-hard hash function like Argon2 or scrypt is far preferable.

I think this is a great recommendation, thank you. I think I picked something "low" for PBKDF2 since there are clients that will have to derive the password every time, which would make sending messages have a delay of 1s or something, or require storing the key somewhere. I'll experiment with the other key derivation functions though.

We now also have access tokens since 2.x, so it may be feasible to ditch the password altogether. I'll look at it when I eventually get to this ticket.

[gardient]

A small question (I have no crypto background): What would happen if two people used (e.g. by accident) the same topic with a different password? Would that be possible? I guess they wouldn't receive the other person's notification and only their own (because they can only decipher their own message)?! Would they get an error message of an attempted notification/wrong password?

@blewsky according to the "General Flow" in #69 (comment) you would get a notification with (encrypted message)

This would only protect content, not the topic itself, you could have a combined encrypted and unencrypted topic even.

also, slightly interesting tidbit, there is nothing really stopping you from creating a custom client and using this with the current setup, the only thing you would miss is the native Encoding header support

[MzHub]

What is the relation here to the Web Push Payload Encryption?

I gave an application server https://ntfy.sh/mytopic as the Web Push endpoint instead of the browser's generated endpoint URL. This resulted in receiving encrypted binary blobs in the ntfy Android app.

Seems to me like the Android app could act like a browser and have its own keys (per subscription), while the CLI or whatever would act as the application server and have its own keys. Also using Web Push libraries would avoid rolling own crypto.

If I'm on the wrong track here I'm interested in learning more about what the difference is.

[SilverBut]

I'm happy to see E2E as a feature to ensure every content being pushed is only visible for the push source and the certain recipients, and not readable for either some vendor's push server (like APNs or Firebase) or temporary push server. Additionally, I think passphrase instead of key is better, because generate a passphrase requires zero knowledge and passphrase can be shared easily. Generate key requires more work, and exchange key content offline is harder.

Thus I think JWE or GPG is good enough. As for Web Push Payload Encryption, it is a pre-HTTPS-everywhere tech, more about preventing insecure push providers using HTTP to send your content, as it stated in the blog:

browser chooses which push provider will be used to actually deliver the payload,..., HTTPS can only guarantee that no one can snoop on the message in transit to the push service provider. Once they receive it, they are free to do what they like, including re-transmitting the payload to third-parties or maliciously altering it to something else. To protect against this we use encryption to ensure that push services can't read or tamper with the payloads in transit.

[Fmstrat]

Seems like much of this could be overkill. A simpler way is to treat it like every other method (Signal, Matrix, etc). Provide a way to send a push that tells the client to check a predetermined URL via HTTPS using its bearer token. The user can then determine if they wish to do E2E, but this would be enough to keep Google's eyes out.

• Add setting in client for custom URL and access token
• Script sends push with "check URL" command
• Client checks URL for latest message and displays the notification.

This is basically what I do for Matrix API from a shell script for pushes via Element/Matrix.

[MzHub]

As for Web Push Payload Encryption, it is a pre-HTTPS-everywhere tech, more about preventing insecure push providers using HTTP to send your content, as it stated in the blog

My reading is that it does not state that Web Push encryption protects from insecure push providers using HTTP.

First they assume everyone uses HTTPS, but "HTTPS can only guarantee that no one can snoop on the message in transit to the push service provider."

Then they go on to explain that HTTPS isn't End-to-End Encryption, and E2EE is needed to prevent Push Providers from snooping on messages, which is why they added E2EE to Web Push.

---

Part of the reason I'm asking if Web Push Payload Encryption has been considered is because it is an existing solution widely used in production already.

Another reason is that while people claim "Web Push compatibility", there can not be any meaningful Web Push compatibility without clients that can decrypt the messages.

Ntfy is almost Web Push compatible. I've had third party services (application servers I do not own) send push messages into my test endpoint. The only missing piece is decryption.

[CyberShadow]

HTTPS can only guarantee that no one can snoop on the message in transit to the TLS terminator. This could be Cloudflare, a corporate security proxy, an evil MITM proxy whose operator has the private key of some TLS certificate trusted by your device, etc. HTTPS is not enough by itself.

[cmj2002]

Any updates for this feature?

[dtinth]

I’d like to suggest NaCl for consideration as an encryption/decryption library. In JavaScript TweetNaCl.js can be used, and for Android Lazysodium can be used. They support both symmetric and asymmetric encryption.

[7heo]

It should be possible to e2e encrypt messages, like this:

NTFY_PASSWORD=... ntfy publish --encrypt mytopic my encrypted message

Or this

echo my encrypted message | <something> | curl -T- -H "Encrypted: yes" ntfy.sh/mytopic

The message should look like this:

{
  "id": "..",
  "type": "message",
  "encryption": "aes-128-gcm",
  "message": "ZnNkZmRzZnNkZmZzZGtqZmRzamZsc2RmCg=="
}

It is important to me that the solution is widely supported and can be easily implemented in all languages. Ideally something like gpg or age, but I doubt such a format exists.

I would like to suggest a two-folds solution for implementing E2E in ntfy; which I believe would greatly ease its implementation.

Sender side

Instead of going

NTFY_PASSWORD=... ntfy publish --encrypt mytopic my encrypted message

Or

echo my encrypted message | <something> | curl -T- -H "Encrypted: yes" ntfy.sh/mytopic

as suggested, why not using:

echo "my plaintext message" | gpg --encrypt --trust-model always --armor -r "$email_or_hash" \
  | curl -d@- ntfy.sh/mytopic

Notice the use of -d@ with curl instead of -T, which results in a POST and not a PUT (I don't know if PUT is currently implemented in ntfy) Never mind, PUT is clearly implemented, it is even in the repository's summary.

Of course, it is necessary to add a recipient flag (-r) per recipient.

This method would literally not require any change to the server code.

Receiver side

On the receiver's end, the message is obtained as usual, but in case the first line is -----BEGIN PGP MESSAGE-----, the message is processed by an appropriate app (OpenKeychain on Android, and PGPro on iOS), so that the resulting clear-text can be displayed; or, if it failed, either the app displays a warning (or error) or silently ignores it, depending on per-topic configuration.

It would be probably good to display an appropriate warning by default, when receiving a PGP encrypted message, if no PGP app is installed on the device.

[bogorad]

Why all this complexity - do as Pushbullet does: if you set a secret on one end, all other endpoints won't be able to read messages unless they know the same secret. Doesn't really matter which encryption library is used, nacl is as good as any. Just mandate a lengthy secret.