Duplicate notifications when APNS receives invalid token

[aquach]

Hi there,

We're seeing people receive duplicate notifications with rpush 2.7.0. For example, here are some incomplete logs with the notification ID 8398879:

[ios] 8398879 sent to <token>
WARNING [ios] Notifications 8398832, 8398833, 8398834, 8398835, 8398836, 8398837, 8398838, 8398839, 8398840, 8398841, 8398842, 8398843, 8398845, 8398846, 8398850, 8398851, 8398852, 8398853, 8398854, 8398855, 8398856, 8398857, 8398859, 8398860, 8398861, 8398862, 8398863, 8398864, 8398865, 8398866, 8398867, 8398868, 8398869, 8398870, 8398871, 8398872, 8398873, 8398874, 8398875, 8398876, 8398877, 8398878, 8398879, 8398880, 8398881, 8398882, 8398883, 8398884, 8398885, 8398886, 8398887, 8398888, 8398889, 8398890, 8398891, 8398892, 8398893, 8398894, 8398895 will be retried due to the failure of notification 8398831.
[ios] 8398879 sent to <token>
WARNING [ios] Notifications 8398833, 8398834, 8398835, 8398836, 8398837, 8398838, 8398839, 8398840, 8398841, 8398842, 8398843, 8398845, 8398846, 8398850, 8398851, 8398852, 8398853, 8398854, 8398855, 8398856, 8398857, 8398859, 8398860, 8398861, 8398862, 8398863, 8398864, 8398865, 8398866, 8398867, 8398868, 8398869, 8398870, 8398871, 8398872, 8398873, 8398874, 8398875, 8398876, 8398877, 8398878, 8398879, 8398880, 8398881, 8398882, 8398883, 8398884, 8398885, 8398886, 8398887, 8398888, 8398889, 8398890, 8398891, 8398892, 8398893, 8398894, 8398895, 8398896, 8398897, 8398898, 8398899, 8398900, 8398901, 8398902, 8398903, 8398904, 8398905, 8398906, 8398907, 8398908, 8398909, 8398910, 8398911, 8398912, 8398913, 8398914, 8398915, 8398916, 8398917, 8398918 will be retried due to the failure of notification 8398832.
[ios] 8398879 sent to <token>
WARNING [ios] Notifications 8398835, 8398836, 8398837, 8398838, 8398839, 8398840, 8398841, 8398842, 8398843, 8398845, 8398846, 8398850, 8398851, 8398852, 8398853, 8398854, 8398855, 8398856, 8398857, 8398859, 8398860, 8398861, 8398862, 8398863, 8398864, 8398865, 8398866, 8398867, 8398868, 8398869, 8398870, 8398871, 8398872, 8398873, 8398874, 8398875, 8398876, 8398877, 8398878, 8398879, 8398880, 8398881, 8398882, 8398883, 8398884, 8398885, 8398886, 8398887, 8398888, 8398889, 8398890, 8398891, 8398892, 8398893, 8398894, 8398895 will be retried due to the failure of notification 8398834.
[ios] 8398879 sent to <token>
WARNING [ios] Notifications 8398836, 8398837, 8398838, 8398839, 8398840, 8398841, 8398842, 8398843, 8398845, 8398846, 8398850, 8398851, 8398852, 8398853, 8398854, 8398855, 8398856, 8398857, 8398859, 8398860, 8398861, 8398862, 8398863, 8398864, 8398865, 8398866, 8398867, 8398868, 8398869, 8398870, 8398871, 8398872, 8398873, 8398874, 8398875, 8398876, 8398877, 8398878, 8398879, 8398880, 8398881, 8398882, 8398883, 8398884, 8398885, 8398886, 8398887, 8398888, 8398889, 8398890, 8398891, 8398892, 8398893, 8398894, 8398895, 8398927 will be retried due to the failure of notification 8398835.
[ios] 8398879 sent to <token>
WARNING [ios] Notifications 8398837, 8398838, 8398839, 8398840, 8398841, 8398842, 8398843, 8398845, 8398846, 8398850, 8398851, 8398852, 8398853, 8398854, 8398855, 8398856, 8398857, 8398859, 8398860, 8398861, 8398862, 8398863, 8398864, 8398865, 8398866, 8398867, 8398868, 8398869, 8398870, 8398871, 8398872, 8398873, 8398874, 8398875, 8398876, 8398877, 8398878, 8398879, 8398880, 8398881, 8398882, 8398883, 8398884, 8398885, 8398886, 8398887, 8398888, 8398889, 8398890, 8398891, 8398892, 8398893, 8398894, 8398895, 8398927, 8398942, 8398943, 8398944, 8398945, 8398946, 8398947, 8398948, 8398949, 8398950, 8398951, 8398952, 8398953, 8398954, 8398955, 8398956, 8398957, 8398958, 8398959 will be retried due to the failure of notification 8398836.

It looks like the APNS sender is sending a whole batch of notifications, but one notification in the batch is failing because its token is invalid. My understanding is that this causes APNS to shut off the connection, so rpush assumes that all notifications sent after the failing notification have not been sent and queues them for retry (hence the "will be retried due to the failure of notification" message).

For some reason, that assumption doesn't seem to be holding up, since we have multiple reports of users receiving the same notification 4-5 times, and as far as I can tell it's because they're in a batch of notifications sent that had a failure earlier in the batch.

Are duplicate notifications expected behavior? If yes, are there configuration settings that can reduce the incidence of these issues? Would decreasing the batch_size help? Is there a way to send notifications and sleep after each one to check for errors before moving on to the next one?

[aquach]

Alternatively, since the errors are all because the token is invalid, is there a way to get updates on when tokens are invalidated fast enough so that we don't have to find out when we send those notifications?

[thuss]

I'm seeing the same issue with the latest published 2.7.0 gem. Users are occasionally receiving duplicate notifications even though we only create a single entry in the rpush_notifications table.

[walidvb]

I'm also frequently seeing duplicates notifications sent, not sure how to go about that.
Any progress or fixes on this issue?

[accessd]

Same problem. From logs: Notifications 631545, 631546, ..., 632174, 632175 will be retried due to the failure of notification 631544. Notification 631544 failed with Invalid device token (8) error.

[gnumarcelo]

Hi, is this still an issue?

[walidvb]

haven't looked at my logs in ages, happy to close and reopen if it comes up again for my part.

[max-did-it]

Have the same issue with FCM.