Consider Including a Matrix Gateway endpoint as part of ntfy

[MayeulC]

Hello, this is kind of awkward to ask as the use-case is quite specific, but at the same time it should be fairly self-contained.

Rationale

The core issue is using UnifiedPush in conjunction with a Matrix Client and Server. The Matrix Client will pass a push URL to the Matrix server. However, the Matrix spec mandates some endpoints for push servers: https://spec.matrix.org/v1.2/push-gateway-api/

Basically, the spec mandates the use of the /_matrix/push/v1/notify path.

Therefore, one has to use an intermediate gateway before forwarding requests to ntfy. When users do not use their own, clients use default ones, which led to rate-limiting: #144

Benefit to users

Currently, users can host a gateway themselves through the common-proxies project, or some other rewrite proxy.

That's one more step for users, who might feel discouraged from doing so.
Including that gateway would increase privacy, for users of the public ntfy instance, but even more so for users who host their own homeserver and ntfy instance, as push messages do not need to travel through third-party gateways.

Needed changes

The set of features to support is relatively simple:

• Answer

   GET /_matrix/push/v1/notify

  With

  HTTP/1.1 200 OK
  
  {"unifiedpush":{"gateway":"matrix"}}

• Handle Pushes on that endpoint. From the spec:

  POST /_matrix/push/v1/notify
  
  {
  "notification": {
    "content": {
      "body": "I'm floating in a most peculiar way.",
      "msgtype": "m.text"
    },
    "counts": {
      "missed_calls": 1,
      "unread": 2
    },
    "devices": [
      {
        "app_id": "org.matrix.matrixConsole.ios",
        "data": {},
        "pushkey": "https://ntfy.sh/exampleTopicID?up=1",
        "pushkey_ts": 12345678,
        "tweaks": {
          "sound": "bing"
        }
      }
    ],
    "event_id": "$3957tyerfgewrf384",
    "prio": "high",
    "room_alias": "#exampleroom:matrix.org",
    "room_id": "!slw48wfj34rtnrf:example.com",
    "room_name": "Mission Control",
    "sender": "@exampleuser:matrix.org",
    "sender_display_name": "Major Tom",
    "type": "m.room.message"
  }
  }

  That should be forwarded to the topic in json_body["notification"]["devices"][0]["pushkey"]. In this case, exampleTopicID?up=1.
• Answer that push with:

  HTTP/1.1 200 OK
  
  {"rejected":[]}

  Technically one could reject events for some topics that have never been listened to, or were deleted from the app, but that sounds a bit alien to the way ntfy works.

That's all. This should only be necessary until the Matrix Spec Change 2970 lands to delete that requirement and allow arbitrary paths, so that's your call.

Impact assessment

One more thing to expect: on the public ntfy instance, FluffyChat's and schildichat's public gateways wouldn't be used anymore, so individual Matrix servers would be subjected to rate-limiting instead, probably Matrix.org, kde.org and mozilla.org, as they are the biggest I can think of right now.

On the other hand, it makes it easier to single out abusive Matrix homeservers without locking users out. Moreover, as more clients appear (Element will soon support UP), more gateways will appear.

It might also make it easier to rate-limit matrix separately while the MSC isn't merged.

[p1gp1g]

For information, that's the functions that does it for NextPush :
https://github.com/UP-NextPush/server-app/blob/main/lib/Controller/UnifiedPushProviderController.php#L374-L420

[binwiederhier]

Sooo... I can see why this is awkward.

You're asking me to put more load on the ntfy.sh server to support an unrelated feature, and introduce a potential security issue by letting the ntfy server make HTTP requests to arbitrary servers (see #93).

I am sympathetic to the cause, yet you have to understand my hesitation. ntfy is a pubsub system primarily for push notifications from scripts and such. It's not a Matrix chat relay/gateway or anything Matrix specific.

[p1gp1g]

I understand your hesitation since it is not directly related to the core of the project :)

Just to make it clear:
We are not asking for a public gateway (ntfy won't do any additional request [1][2]), just an endpoint (/_matrix/push/v1/notify) where the subject is read at body["notification"]["devices"][0]["pushkey"] [3] instead of the path [4]. So notification will do [matrix-server] -> [ntfy] instead of [matrix-server] -> [gateway] -> [ntfy]

For instance, the following request would do the same thing than posting it to https://ntfy.sh/exampleTopicID?up=1

POST /_matrix/push/v1/notify

{
"notification": {
  "content": {
    "body": "I'm floating in a most peculiar way.",
    "msgtype": "m.text"
  },
  "counts": {
    "missed_calls": 1,
    "unread": 2
  },
  "devices": [
    {
      "app_id": "org.matrix.matrixConsole.ios",
      "data": {},
      "pushkey": "https://ntfy.sh/exampleTopicID?up=1",
      "pushkey_ts": 12345678,
      "tweaks": {
        "sound": "bing"
      }
    }
  ],
  "event_id": "$3957tyerfgewrf384",
  "prio": "high",
  "room_alias": "#exampleroom:matrix.org",
  "room_id": "!slw48wfj34rtnrf:example.com",
  "room_name": "Mission Control",
  "sender": "@exampleuser:matrix.org",
  "sender_display_name": "Major Tom",
  "type": "m.room.message"
}
}

Once again, I understand if you do not want to implement it.

[1] Which would be totally unrelated to this project ^^
[2] So this won't add any more load nor introduce potential SSRF :)
[3] To be exact, ["pushkey"] for all device in body["notification"]["devices"]
[4] See https://github.com/UP-NextPush/server-app/blob/main/lib/Controller/UnifiedPushProviderController.php#L374-L420

[binwiederhier]

Ohhh I see. Yeah I can totally do that.

[MayeulC]

Thanks @p1gp1g for clarifying.

Also, while it would be nice to have it on the public ntfy.sh instance, I think it brings more value for self-hosted instances, as users don't have to fiddle with setting up a gateway or rewrite proxy themselves, which they can get wrong.

[binwiederhier]

I thought this was a fun 30min thing to do and here's the result: #326

This implementation is incomplete and has issues:

• It doesn't return the correct result as per spec (https://spec.matrix.org/v1.2/push-gateway-api/). It just returns the ntfy response. This is solvable but annoying.
• It does not handle all devices in the device array, but only the first one. This is technically solvable, but would mean having to translate 1 request to N requests. This is something that I have avoided at all cost so far, because it mucks with rate limiting and such. I am not sure if I want to allow this. Is is realistic that there are more than 1 device in this array?

[p1gp1g]

I thought this was a fun 30min thing to do and here's the result: #326

I thought it too, but it might take more time depending on the internal architecture :/

• It doesn't return the correct result as per spec (https://spec.matrix.org/v1.2/push-gateway-api/). It just returns the ntfy response. This is solvable but annoying.

Without the response {"rejected": []} (or {"rejected": ["pushkey_exmpl"]} which seems impossible with ntfy, but that's not very important), the matrix homeserver will probably spam the endpoint.

Also, without the discovery response (bellow), clients won't know there is a gateway and won't use it.

GET /_matrix/push/v1/notify

{"unifiedpush":{"gateway":"matrix"}}

• It does not handle all devices in the device array, but only the first one. This is technically solvable, but would mean having to translate 1 request to N requests. This is something that I have avoided at all cost so far, because it mucks with rate limiting and such. I am not sure if I want to allow this. Is is realistic that there are more than 1 device in this array?

It seems all homeservers send only one device at this moment. I don't know if any plan to change this behavior

• Synapse : https://github.com/matrix-org/synapse/blob/a164a46038b0e51142781619db0e6dec8e0c2aaa/synapse/push/httppusher.py#L362
• Dendrite : https://github.com/matrix-org/dendrite/blob/f05ce478f05dcaf650fbae68a39aaf5d9880a580/userapi/util/notify.go#L50
• Ruma (Conduit) : https://github.com/ruma/ruma/blob/a47c8f08d34cd09fbc583a11bbbe7b592e05592a/crates/ruma-push-gateway-api/src/send_event_notification.rs#L440

[MayeulC]

Hey, I was not expecting you to have a look at it so soon. Thanks a lot!

Here are my two cents (mostly the same as @p1gp1g's message):

• Indeed, a GET should return that {"unifiedpush":{"gateway":"matrix"}} response. It's less complex to add this to the reverse proxy than de-serializing the json, but it should be fairly easy to implement inside ntfy.
• For now, allowing only 1 device should be fine, I don't foresee this changing before MSC2970 lands.
• I think it's better to return {"rejected":[]} in every case than returning the normal ntfy response. Ideally, that shouldn't be returned when rate-limitting happens, but I am not sure what should be returned in that case.
  Edit: I now see why it's more complex than anticipated, the response is only generated for an error. I can imagine cheating by returning an error code "200" and a body, but that requires adjusting the handle function in any case.

[binwiederhier]

I think I can do the response bit, as long as I don't have to implement 1-to-N I'm okay with the hacks. Just thought I'd ask if that's okay before proceeding.

[p1gp1g]

Right now, I'd say the discovery response + the {"rejected":[]} response works perfectly fine. But we have to keep in mind that homeservers might change their behavior in the future :/

[binwiederhier]

I forgot the GET but that'll be easy. It's still ugly but here's the behavior:

Entirely invalid request

$ curl -s -XPOST -d 'this is invalid' localhost:2586/_matrix/push/v1/notify
{"code":40019,"http":400,"error":"invalid request: Matrix JSON invalid","link":"https://ntfy.sh/docs/publish/#matrix-gateway"}

Log:

2022/06/14 20:42:29 DEBUG 127.0.0.1 HTTP POST /_matrix/push/v1/notify Dispatching request
2022/06/14 20:42:29 DEBUG 127.0.0.1 HTTP POST /_matrix/push/v1/notify Connection closed with HTTP 400 (ntfy error 40019): invalid request: Matrix JSON invalid

Valid request:

$ curl -s -XPOST -T request.json localhost:2586/_matrix/push/v1/notify     
{"rejected":[]}

Log:

2022/06/14 20:42:46 DEBUG 127.0.0.1 HTTP POST /_matrix/push/v1/notify Dispatching request
2022/06/14 20:42:46 DEBUG 127.0.0.1/exampleTopicID/s3Zr6WymX1P7 Received message: event=message, body=698 byte(s), delayed=false, firebase=false, cache=true, up=true, email=
2022/06/14 20:42:46 TRACE 127.0.0.1/exampleTopicID/s3Zr6WymX1P7 Message body: {
  "id": "s3Zr6WymX1P7",
  "time": 1655253766,
  "event": "message",
  "topic": "exampleTopicID",
  "message": "{\n\"notification\": {\n  \"content\": {\n    \"body\": \"I'm floating in a most peculiar way.\",\n    \"msgtype\": \"m.text\"\n  },\n  \"counts\": {\n    \"missed_calls\": 1,\n    \"unread\": 2\n  },\n  \"devices\": [\n    {\n      \"app_id\": \"org.matrix.matrixConsole.ios\",\n      \"data\": {},\n      \"pushkey\": \"http://127.0.0.1:2586/exampleTopicID?up=1\",\n      \"pushkey_ts\": 12345678,\n      \"tweaks\": {\n        \"sound\": \"bing\"\n      }\n    }\n  ],\n  \"event_id\": \"$3957tyerfgewrf384\",\n  \"prio\": \"high\",\n  \"room_alias\": \"#exampleroom:matrix.org\",\n  \"room_id\": \"!slw48wfj34rtnrf:example.com\",\n  \"room_name\": \"Mission Control\",\n  \"sender\": \"@exampleuser:matrix.org\",\n  \"sender_display_name\": \"Major Tom\",\n  \"type\": \"m.room.message\"\n}\n}\n"
}
2022/06/14 20:42:46 TRACE 127.0.0.1/exampleTopicID/s3Zr6WymX1P7 No stream or WebSocket subscribers, not forwarding

Valid Matrix message, but with invalid push key

$ cat request.json | jq '.notification.devices[0].pushkey'
"http://not-the-base-url-127.0.0.1:2586/exampleTopicID?up=1"

$ curl -s -XPOST -T a localhost:2586/_matrix/push/v1/notify
{"rejected":["http://not-the-base-url-127.0.0.1:2586/exampleTopicID?up=1"]}

Log:

2022/06/14 20:43:01 DEBUG 127.0.0.1 HTTP POST /_matrix/push/v1/notify Dispatching request
2022/06/14 20:43:01 DEBUG Matrix message with push key http://not-the-base-url-127.0.0.1:2586/exampleTopicID?up=1 rejected: invalid request: Matrix JSON invalid