[ads] Intermittent crash while refilling confirmation tokens during Later captcha testing

[GeetaSarvadnya]

Description

Found while testing #37790
Brave crashes sometimes while Token refill during Later captcha testing

Steps to Reproduce

1. Clean profile 1.65.124 staging env
2. Launch brave using brave.exe --enable-logging=stderr --vmodule="*/bat-native-ledger/*"=9,"*/brave_rewards/*"=9,"*/bat-native-ads/*"=9,"*/bat-native-confirmations/*"=9,"*/brave_ads/*"=9,"*/brave_user_model/*"=9 --rewards=staging=true,debug=true
3. Enable rewards and wait for Ads initialization and 50 paymnet token refill
4. View an Ad (I have viewed and clicked on Ad)
5. 48 payment tokens remined via console logs
6. Schedule a captcha
7. Click on Up and Down arrow 30 times, captcha notification appears and click on Later button
8. 18 payment tokens remained which is below the min threshold, after sometime I have got logs as below

[2716:12796:0429/143513.069:INFO:refill_confirmation_tokens.cc(197)] Captcha is required to refill confirmation tokens
[2716:12796:0429/143513.070:VERBOSE1:user_rewards.cc(122)] Failed to refill confirmation tokens

9. Keep watching the logs, I am seeing Refill request failed about 4 or 5 times
   At some point browser crashed

Actual result:

Brave crashes sometimes while Token refill during Later captcha testing

Uploaded Crash Report ID: 19f11200-505c-df0b-0000-000000000000
Uploaded Crash Report ID: 5a7e1000-505c-df0b-0000-000000000000

Expected result:

Should not be any crash

Reproduces how often:

Intermittent issue

Brave version (brave://version info)

Brave	1.65.124 Chromium: 124.0.6367.91 (Official Build) (64-bit)
Revision	af639d0b172be01f63a6b8b0187ba24870fa04f7
OS	Windows 10 Version 22H2 (Build 19045.4291)

Version/Channel Information:

• Can you reproduce this issue with the current release? Yes
• Can you reproduce this issue with the beta channel? Not sure
• Can you reproduce this issue with the nightly channel? Not sure

Other Additional Information:

• Does the issue resolve itself when disabling Brave Shields? NA
• Does the issue resolve itself when disabling Brave Rewards? NA
• Is the issue reproducible on the latest version of Chrome? NA

Miscellaneous Information:

cc: @brave/qa-team @rebron @tmancey

[tmancey]

Closing as we believe this should be resolved by #37957, after tersting 37957 if the issue persists can you please reopen this issue. thanks

[GeetaSarvadnya]

The issue is verified on the following build, the issue still persists hence re-opening the issue.

Brave | 1.67.58 Chromium: 124.0.6367.118 (Official Build) nightly (64-bit)
-- | --
Revision | 7feaff7da4011f6d97f4d55137f8aaf2b3052b7c
OS | Windows 10 Version 22H2 (Build 19045.4291)

Crash Id reports:

Uploaded Crash Report ID: 66a40200-ea91-e80b-0000-000000000000
Uploaded Crash Report ID: f2660200-ea91-e80b-0000-000000000000

[aseren]

I couldn't reproduce the issue with a fresh master branch.

Unfortunately crashes from @GeetaSarvadnya, do not contain symbols so it hard to tell the reason.

Stack trace from Crash report f2660200-ea91-e80b-0000-000000000000 shows that it is not a real crash but just a Dump Without Crashing, so I believe it is not really connected to the issue.

[ 00 ] crash_reporter::DumpWithoutCrashing() ( crashpad.cc:269 )
[ 01 ] 0x7ffe71ac455f
[ 02 ] 0x7ffe79ca99ba
[ 03 ] 0x7ffe74a81b07
[ 04 ] 0x7ffe714d82fa
[ 05 ] 0x7ffe7ac2b3f0
[ 06 ] 0x7ffe79ca9a62
[ 07 ] 0x7ffe79ca99ba
[ 08 ] 0x7ffe74a81af5
[ 09 ] 0x7ffe74a81ead
[ 10 ] 0x7ffe6d13c1e6
[ 11 ] 0x7ffe79ca9be7
[ 12 ] 0x7ffe6edecaa2
[ 13 ] 0x7ffe719e84a7
[ 14 ] 0x7ffe6f712f77
[ 15 ] 0x7ffe6d2f8153
[ 16 ] 0x7ffe6d131ded
[ 17 ] BaseThreadInitThunk
[ 18 ] RtlUserThreadStart

[aseren]

I was able to reproduce the crash locally. It crashed with the message:

[94961:83723:0502/223032.083620:FATAL:refill_confirmation_tokens.cc(302)] Check failed: !retry_timer_.IsRunning(). 

[tmancey]

Crash Reason:

Following system resumption, an attempt is made to refill confirmation tokens:

[22055:259:0503/140025.271338:VERBOSE1:user_rewards.cc(114)] Refill confirmation tokens
[22055:259:0503/140025.273437:VERBOSE1:refill_confirmation_tokens.cc(105)] Request signed tokens

However, this attempt fails due to:

[22055:259:0503/140025.304867:VERBOSE6:refill_confirmation_tokens.cc(121)] URL Response:
  URL: https://mywallet.ads.bravesoftware.com/v3/confirmation/token/2984f3ce-7c72-52a7-8c37-5fdbe969a6f6
  Response Status Code: 429
  Response: {"error":"Number of allowed tokens exceeded","statusCode":429}
[22055:259:0503/140025.305090:INFO:refill_confirmation_tokens.cc(128)] Failed to request signed tokens
[22055:259:0503/140025.305275:VERBOSE1:user_rewards.cc(122)] Failed to refill confirmation tokens

Following this, a timer is initiated to retry refilling the confirmation tokens:

[22055:259:0503/140025.305454:VERBOSE1:user_rewards.cc(128)] Retry refilling confirmation tokens in 0 hours, 21 minutes, 29 seconds at 14:21:54.304

Additionally, within the Chromium codebase, the same timer is restarted upon power resumption causing the timer to restart twice causing a crash.

cc @GeetaSarvadnya @btlechowski @aseren

[kjozwiak]

The above requires 1.65.129 or higher for 1.65.x verification 👍

[GeetaSarvadnya]

Verification PASSED on

Brave | 1.65.130 Chromium: 124.0.6367.159 (Official Build) (64-bit)
-- | --
Revision | 67843557ad274aba4b206391773c789c174b7fad
OS | Windows 10 Version 22H2 (Build 19045.4291)

• Verified the STR from [ads] Intermittent crash while refilling confirmation tokens during Later captcha testing #37934 (comment) and ensured that there is no crash when refill token request failed
• Verified the issue couple of times and ensured that there is no crash