Not receiving WebPush notifications on an endpoint to a PHP app

[mbootsman]

Steps to reproduce the problem

1. Create subscription with the API and use an URL of the PHP app as endpoint
2. Mention the account that has the subscription
3. Use the app to receive the WebPush notification
4. Nothing is received...

Expected behaviour

PHP app on the endpoint should receive a POST request with the WebPush notification

Actual behaviour

Nothing

Detailed description

I have tested this on my own server toot.re (4.2.8) and mastodon.social (v4.3.0-nightly.2024-03-19)

Endpoint is publicly accessible. Checked this with various online tools.

Mastodon instance

No response

Mastodon version

No response

Technical details

If this is happening on your own Mastodon server, please fill out those:

• Ruby version: (from ruby --version, eg. v3.1.2)
  ruby 3.0.4p208 (2022-04-12 revision 3fa771dded) [x86_64-linux]

• Node.js version: (from node --version, eg. v18.16.0)
  v16.20.2

[ThisIsMissEm]

What does the URL you're using look like?

[mbootsman]

What does the URL you're using look like?

https://my.domain.name/remindme/pushme.php

[ThisIsMissEm]

On your own server, do you see anything in the logs, sidekiq dashboard, or in the rails console via Web::PushSubscription.find_by_endpoint("...")?

[ThisIsMissEm]

In the Notifications tab/column, if you go to the settings, are any of these enabled as "for desktop"?

i think this may be designed such that there's only one setting, not independent per push subscription.

[mbootsman]

I found my subscription with the ruby console:

irb(main):001:0> Web::PushSubscription.find_by_endpoint("https://my.domain.name/remindme/pushme.php")
=> 
#<Web::PushSubscription:0x00007f7c79339418
 id: 7768,
 endpoint: "https://my.domain.name/remindme/pushme.php",
 key_p256dh: "--removed--",
 key_auth: "--removed--",
 data: {"alerts"=>{"mention"=>"1"}},
 created_at: Fri, 22 Mar 2024 14:57:56.934082000 UTC +00:00,
 updated_at: Fri, 22 Mar 2024 14:57:56.934082000 UTC +00:00,
 access_token_id: --removed--,
 user_id: 5>

Can you tell me in what logs I can look to find the notifications?

[mbootsman]

In the Notifications tab/column, if you go to the settings, are any of these enabled as "for desktop"?

i think this may be designed such that there's only one setting, not independent per push subscription.

Yes, these are all enabled.

[ThisIsMissEm]

I found my subscription with the ruby console:

irb(main):001:0> Web::PushSubscription.find_by_endpoint("https://my.domain.name/remindme/pushme.php")
=> 
#<Web::PushSubscription:0x00007f7c79339418
 id: 7768,
 endpoint: "https://my.domain.name/remindme/pushme.php",
 key_p256dh: "--removed--",
 key_auth: "--removed--",
 data: {"alerts"=>{"mention"=>"1"}},
 created_at: Fri, 22 Mar 2024 14:57:56.934082000 UTC +00:00,
 updated_at: Fri, 22 Mar 2024 14:57:56.934082000 UTC +00:00,
 access_token_id: --removed--,
 user_id: 5>

Can you tell me in what logs I can look to find the notifications?

okay, and this is what I'd expect (I'd actually originally missed the data blob on the Web::PushSubscription model)

[ThisIsMissEm]

You'd see these as dead jobs in Sidekiq, but I'm not 100% sure. /sidekiq/morgue on the mastodon instance.

[mbootsman]

I see one item in sidekiq/retries, a few minutes after I triggered a notification:

Web::PushNotificationWorker | 7768, 48338 | OpenSSL::PKey::EC::Point::Error: EC_POINT_bn2point: invalid encoding

Edit: And another one, with different arguments when mentioning my account. Could this be the error, preventing Mastodon from send the push notification?

[ThisIsMissEm]

What is your key_p256dh value? (it should be unique per subscription iirc), the subscription[keys][p256dh] value must be URL Safe Base64 encoded, perhaps the issue is that when you're creating the push subscription the value isn't received correctly?

In modern ruby, the key_p256dh value goes through essentially the following code:

decodedKey = Base64.urlsafe_decode64(key_p256dh)

group = OpenSSL::PKey::EC::Group.new('prime256v1')
client_public_key_bn = OpenSSL::BN.new(decodedKey, 2)
client_public_key = OpenSSL::PKey::EC::Point.new(group, client_public_key_bn)

So if that throws with your key_p256dh value, then you would get an error around OpenSSL::PKey::EC::Point::Error: EC_POINT_bn2point: invalid encoding — tbh, on Push Subscription creation the key_p256dh value should probably be validated (decoded, and tried to be created on the given elliptical curve (EC), and if that fails, then the subscription should not be created.

What's your PHP code for creating the subscription?

References:

• https://github.com/mastodon/mastodon/blob/main/app/models/web/push_subscription.rb#L30
• https://github.com/ClearlyClaire/webpush/blob/master/lib/webpush/encryption.rb#L8

[mbootsman]

This is the $post_data I post to the instance subscriptions API (/api/v1/push/subscription).

$post_data = array(
        "subscription" => array(
            "endpoint" => $env["endpoint"], // Endpoint URL of PHP script
            "keys" => array(
                "p256dh" => base64_encode($env["public_key"]), // Base64 encoded string of a public key from a ECDH keypair using 
                "auth" => $env['subscription_auth']  // Auth secret. Base64 encoded string of 16 bytes of random data. 
            )
        ),
        "data" => array(
            "alerts" => array(
                "mention" => true // Receive notifications for mentions
            )
        )
    );

$env['public_key'] is generated here: https://www.stephane-quantin.com/en/tools/generators/vapid-keys

[ThisIsMissEm]

Okay, so I can't look at how that key is generated, but my understanding was that the vapid key was on the sending side, not the receiving, and that the receiving side the key is for encryption. Did you look at the php webpush library linked from that article? It uses several methods from a Jose package around base64 encoding without padding.

[ThisIsMissEm]

https://github.com/web-push-libs/web-push-php/tree/master

[TomSmoot]

FWIW I did the same thing in PHP, set up endpoints to request and receive push notifications. I set up my keys and got a response with a push subscription ID. It seems my subscription response was truncated though and 100% positive my push handler endpoint is not receiving any traffic. Gonna try requesting a new subscription and see if I can reproduce it or not. But for @mbootsman I used PuTTY keygen to create the pem/pub files and grabbed them with this:

// Load private key from .pem file
$private_key_file = 'path';
$private_key = file_get_contents($private_key_file);

// Load public key from .pub file
$public_key_file = 'path';
$public_key = file_get_contents($public_key_file);

// Encode public key in base64
$public_key_base64 = base64_encode($public_key);

[ThisIsMissEm]

Does the .pub file have any headers? Is it already base64 url unsafe encoded?

The web push subscription endpoint will accept invalid arguments. I might spend some time improving this today.

[TomSmoot]

Ooo... I think my .pub file has headers. They both should, I didn't alter any PuTTY output after using the random keygen. And it gives key type options, I used the p265 option so maybe I don't need to encode. I didn't look at the header when I sent the request. Good to know it accepts my garbage tho. When I was reading docs, I read them as bad subscriptions might be due to instance admin issues. (love my admin)

FWIW I really want these subscriptions in an effort to reduce load on instance servers, especially the smaller ones. I imagine when this process goes sideways, too many opt to poll servers every X seconds, which probably isn't ideal on both ends. In that aspect, optional home feed update notifications would be very welcome.