Redis-server abort in dlmalloc

[orangemocha]

Split from #167 (comment), as this is a different issue.

As first reported by @nmehlei:

Yes we upgraded to 2.8.19.1 on Saturday morning and already experienced one crash (Saturday evening).

Details:

Application Error Faulting application name: redis-server.exe, version: 0.0.0.0, time stamp: 0x5547a2d5
Faulting module name: redis-server.exe, version: 0.0.0.0, time stamp: 0x5547a2d5
Exception code: 0xc0000409
Fault offset: 0x0000000000032de0
Faulting process id: 0xf9c
Faulting application start time: 0x01d08a4e9d574822
Faulting application path: C:\Program Files\Redis\redis-2.8.19.1\redis-server.exe
Faulting module path: C:\Program Files\Redis\redis-2.8.19.1\redis-server.exe
Report Id: f0e8be3a-f680-11e4-80bf-000d3a20bbd9
Faulting package full name:
Faulting package-relative application ID:

I investigated the issue through Windows Error Reporting mini-dumps (BEX64 Bucket 81858187473). All crash reports share this common call stack at the moment of the crash:

Child-SP RetAddr Call Site

00 000000000013f610 000000014002a73b redis_server!abort+0x34 [f:\dd\vctools\crt\crtw32\misc\abort.c @ 88]
01 000000000013f640 00000001400be813 redis_server!dlmalloc+0x3ab [d:\redis\msot1\src\win32_interop\win32_dlmalloc.c @ 4647]

.. indicating a memory corruption. No additional clues can be gathered from the mini-dumps. Full dumps requested from user site.

[orangemocha]

@nmehlei: thanks for your help. As previously discussed, a full dump would be the best aid in narrowing down this issue. Also, please make sure to save the redis logs. Even if they probably won't contain any direct information about the error, we might find some correlation between multiple instances of the failure. In fact, if you have the log from last Saturday's failure, that would be an interesting first data point.

[bruceliu2008]

Hello, @orangemocha, may I know what is the issue for crash without any symptom?
This is my crash issue, in my redis.log there is no any error, it just suddenly crash, and after 1 minute, the watcher managed to restart it. But it is just a workaround, it caused my ServiceStack.Redis subscribers stopped as well because it disconnected immediately after the Redis is crashed.

[orangemocha]

Hi @bruceliu2008 , this issue would not present any symptoms, meaning no messages in the logs. So if you experienced a crash without symptoms, it might be this issue, though it's also possible that it could be a different issue.

The information in the event viewer would clarify that. In the event viewer look under Windows Logs \ Application for events with source "Application Error" or "Windows Error Reporting". Those events should contain information that will help us investigate the failure. Can you copy & paste it in here?

Please also ensure that your machine is configured to report crash information. Go to Control Panel\System and Security\Action Center\Problem Reporting Settings and make sure that "Automatically check for solutions and send additional report data, if needed".

As the ultimate troubleshooting aid, you could also enable collection of full-memory dumps.The instructions are here: https://msdn.microsoft.com/en-us/library/windows/desktop/bb787181(v=vs.85).aspx . You can configure it for redis-server.exe only (the article explains how to do so).

Keep in mind that those dumps will be roughly as large on disk as the memory used by redis at the time of the crash.

I will add instructions for this to the readme/wiki.

[bruceliu2008]

Here is the error message in windows event log.
Faulting application name: redis-server.exe, version: 0.0.0.0, time stamp: 0x54edbbf6
Faulting module name: redis-server.exe, version: 0.0.0.0, time stamp: 0x54edbbf6
Exception code: 0xc0000005
Fault offset: 0x000000000002b505
Faulting process id: 0x7904
Faulting application start time: 0x01d08ed98fe44c1f
Faulting application path: c:\redis\bin\redis-server.exe
Faulting module path: c:\redis\bin\redis-server.exe
Report Id: d0f8fe7d-fafa-11e4-80bf-98be9429784b
Faulting package full name:
Faulting package-relative application ID:

[bruceliu2008]

For the WER setting we will enable on next Monday, now it is around midnight in my local time. risky to change my production server's setting now.

[bruceliu2008]

Hi, Any update on this ticket?
We have configured the WER in our servers.
The bad news is that our redis-server.exe is crashed twice and the redis watcher restarted the redis server after the redis-server.exe is dead for 20 minutes last night, which is not acceptable in any production environment.

Thanks,
Bruce

[nmehlei]

"Unfortunately", it didn't crash at our end, so we weren't able to collect any more data. We estimate that our (daily-increasing) traffic will be risen enough in a week or so to trigger those crashes again, which will force us to migrate to linux if it is not fixed until then :/

[bruceliu2008]

Here is our latest report.wer file content.
Version=1
EventType=APPCRASH
EventTime=130764009788562777
ReportType=2
Consent=1
UploadTime=130764072789930463
ReportIdentifier=ce7d5fd3-fd1f-11e4-80bf-98be9429784b
IntegratorReportIdentifier=ce7d5fd2-fd1f-11e4-80bf-98be9429784b
NsAppName=redis-server.exe
Response.BucketId=b8bd7d4d2499fc1eb539d7f8a4aff02b
Response.BucketTable=4
Response.LegacyBucketId=86025357384
Response.type=4
Sig[0].Name=Application Name
Sig[0].Value=redis-server.exe
Sig[1].Name=Application Version
Sig[1].Value=0.0.0.0
Sig[2].Name=Application Timestamp
Sig[2].Value=54edbbf6
Sig[3].Name=Fault Module Name
Sig[3].Value=redis-server.exe
Sig[4].Name=Fault Module Version
Sig[4].Value=0.0.0.0
Sig[5].Name=Fault Module Timestamp
Sig[5].Value=54edbbf6
Sig[6].Name=Exception Code
Sig[6].Value=c0000005
Sig[7].Name=Exception Offset
Sig[7].Value=000000000002b505
DynamicSig[1].Name=OS Version
DynamicSig[1].Value=6.3.9600.2.0.0.272.7
DynamicSig[2].Name=Locale ID
DynamicSig[2].Value=1033
DynamicSig[22].Name=Additional Information 1
DynamicSig[22].Value=d395
DynamicSig[23].Name=Additional Information 2
DynamicSig[23].Value=d3959f151678d37dcc5fd083726ad4bc
DynamicSig[24].Name=Additional Information 3
DynamicSig[24].Value=a27d
DynamicSig[25].Name=Additional Information 4
DynamicSig[25].Value=a27d073b835153799f2360c25db7f06c
UI[2]=c:\redis\bin\redis-server.exe
UI[5]=Check online for a solution (recommended)
UI[6]=Check for a solution later (recommended)
UI[7]=Close
UI[8]=redis-server.exe stopped working and was closed
UI[9]=A problem caused the application to stop working correctly. Windows will notify you if a solution is available.
UI[10]=&Close
LoadedModule[0]=c:\redis\bin\redis-server.exe
LoadedModule[1]=C:\Windows\SYSTEM32\ntdll.dll
LoadedModule[2]=C:\Windows\system32\KERNEL32.DLL
LoadedModule[3]=C:\Windows\system32\KERNELBASE.dll
LoadedModule[4]=C:\Windows\system32\SHLWAPI.dll
LoadedModule[5]=C:\Windows\system32\PSAPI.DLL
LoadedModule[6]=C:\Windows\system32\USER32.dll
LoadedModule[7]=C:\Windows\system32\ADVAPI32.dll
LoadedModule[8]=C:\Windows\system32\SHELL32.dll
LoadedModule[9]=C:\Windows\system32\msvcrt.dll
LoadedModule[10]=C:\Windows\SYSTEM32\combase.dll
LoadedModule[11]=C:\Windows\system32\GDI32.dll
LoadedModule[12]=C:\Windows\SYSTEM32\sechost.dll
LoadedModule[13]=C:\Windows\system32\RPCRT4.dll
LoadedModule[14]=C:\Windows\system32\ws2_32.dll
LoadedModule[15]=C:\Windows\system32\NSI.dll
LoadedModule[16]=C:\Windows\SYSTEM32\SHCORE.dll
LoadedModule[17]=C:\Windows\SYSTEM32\CRYPTBASE.DLL
LoadedModule[18]=C:\Windows\SYSTEM32\bcryptPrimitives.dll
LoadedModule[19]=C:\Windows\system32\mswsock.dll
State[0].Key=Transport.DoneStage1
State[0].Value=1
FriendlyEventName=Stopped working
ConsentKey=APPCRASH
AppName=redis-server.exe
AppPath=c:\redis\bin\redis-server.exe
NsPartner=windows
NsGroup=windows8
ApplicationIdentity=E3EEAAB1B74F642698646ACCC864D688

[bruceliu2008]

Hi All,
We downgrade the server to 2.6.12, it didn't crash since yesterday morning around 8am.

[nmehlei]

I think those versions had other critical bugs but yes maybe they are easier to work around, we're considering downgrading aswell.

[bruceliu2008]

In terms of stability, it should be ok, I have another server already uses it for 73 days without any crash.
it was a stupid idea to upgrade.

[bruceliu2008]

Hello, Any update on this ticket after 6 days?

[orangemocha]

@bruceliu2008 , without a repro or a full dump, it's difficult to make progress on this issue. We are investigating other failures hoping that they could be related.

[nmehlei]

Redis just crashed again

As the full dump is rather large (3 GiB), how should I send you the link? I'd rather not post that link here.

---

Application Error Faulting application name: redis-server.exe, version: 0.0.0.0, time stamp: 0x5547a2d5
Faulting module name: redis-server.exe, version: 0.0.0.0, time stamp: 0x5547a2d5
Exception code: 0xc0000005
Fault offset: 0x0000000000076a2c
Faulting process id: 0x9f8
Faulting application start time: 0x01d0914890c1a1e2
Faulting application path: C:\Program Files\Redis\redis-2.8.19.1\redis-server.exe
Faulting module path: C:\Program Files\Redis\redis-2.8.19.1\redis-server.exe
Report Id: 71fd018c-05e5-11e5-80c1-000d3a20bbd9
Faulting package full name:
Faulting package-relative application ID:

---

Windows Error Reporting Fault bucket , type 0
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: redis-server.exe
P2: 0.0.0.0
P3: 5547a2d5
P4: redis-server.exe
P5: 0.0.0.0
P6: 5547a2d5
P7: c0000005
P8: 0000000000076a2c
P9:
P10:

Attached files:
C:\Windows\Temp\WERFB77.tmp.appcompat.txt
C:\Windows\Temp\WERFC82.tmp.WERInternalMetadata.xml
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_redis-server.exe_6be21d121b71175d17501be6bd60226fa41b448c_0627ca68_cab_497afcaf\memory.hdmp
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_redis-server.exe_6be21d121b71175d17501be6bd60226fa41b448c_0627ca68_cab_497afcaf\triagedump.dmp

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_redis-server.exe_6be21d121b71175d17501be6bd60226fa41b448c_0627ca68_cab_497afcaf

Analysis symbol:
Rechecking for solution: 0
Report Id: 71fd018c-05e5-11e5-80c1-000d3a20bbd9
Report Status: 4
Hashed bucket:

---

Windows Error Reporting Fault bucket , type 0
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: redis-server.exe
P2: 0.0.0.0
P3: 5547a2d5
P4: redis-server.exe
P5: 0.0.0.0
P6: 5547a2d5
P7: c0000005
P8: 0000000000076a2c
P9:
P10:

Attached files:
C:\Windows\Temp\WERFB77.tmp.appcompat.txt
C:\Windows\Temp\WERFC82.tmp.WERInternalMetadata.xml
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_redis-server.exe_6be21d121b71175d17501be6bd60226fa41b448c_0627ca68_cab_497afcaf\memory.hdmp
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_redis-server.exe_6be21d121b71175d17501be6bd60226fa41b448c_0627ca68_cab_497afcaf\triagedump.dmp

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_redis-server.exe_6be21d121b71175d17501be6bd60226fa41b448c_0627ca68_cab_497afcaf

Analysis symbol:
Rechecking for solution: 0
Report Id: 71fd018c-05e5-11e5-80c1-000d3a20bbd9
Report Status: 0
Hashed bucket:

[bruceliu2008]

Guys, I recommend you use linux one. M$ is not trustable in this case.
On 29 May 2015 18:00, "nmehlei" notifications@github.com wrote:

Redis just crashed again

As the full dump is rather large (3 GiB), how should I send you the link?

I'd rather not post that link here.

Application Error Faulting application name: redis-server.exe, version:
0.0.0.0, time stamp: 0x5547a2d5
Faulting module name: redis-server.exe, version: 0.0.0.0, time stamp:
0x5547a2d5
Exception code: 0xc0000005
Fault offset: 0x0000000000076a2c
Faulting process id: 0x9f8
Faulting application start time: 0x01d0914890c1a1e2
Faulting application path: C:\Program
Files\Redis\redis-2.8.19.1\redis-server.exe
Faulting module path: C:\Program
Files\Redis\redis-2.8.19.1\redis-server.exe
Report Id: 71fd018c-05e5-11e5-80c1-000d3a20bbd9
Faulting package full name:

Faulting package-relative application ID:

Windows Error Reporting Fault bucket , type 0
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: redis-server.exe
P2: 0.0.0.0
P3: 5547a2d5
P4: redis-server.exe
P5: 0.0.0.0
P6: 5547a2d5
P7: c0000005
P8: 0000000000076a2c
P9:
P10:

Attached files:
C:\Windows\Temp\WERFB77.tmp.appcompat.txt
C:\Windows\Temp\WERFC82.tmp.WERInternalMetadata.xml

C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_redis-server.exe_6be21d121b71175d17501be6bd60226fa41b448c_0627ca68_cab_497afcaf\memory.hdmp

C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_redis-server.exe_6be21d121b71175d17501be6bd60226fa41b448c_0627ca68_cab_497afcaf\triagedump.dmp

These files may be available here:

C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_redis-server.exe_6be21d121b71175d17501be6bd60226fa41b448c_0627ca68_cab_497afcaf

Analysis symbol:
Rechecking for solution: 0
Report Id: 71fd018c-05e5-11e5-80c1-000d3a20bbd9
Report Status: 4

Hashed bucket:

Windows Error Reporting Fault bucket , type 0
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: redis-server.exe
P2: 0.0.0.0
P3: 5547a2d5
P4: redis-server.exe
P5: 0.0.0.0
P6: 5547a2d5
P7: c0000005
P8: 0000000000076a2c
P9:
P10:

Attached files:
C:\Windows\Temp\WERFB77.tmp.appcompat.txt
C:\Windows\Temp\WERFC82.tmp.WERInternalMetadata.xml

C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_redis-server.exe_6be21d121b71175d17501be6bd60226fa41b448c_0627ca68_cab_497afcaf\memory.hdmp

C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_redis-server.exe_6be21d121b71175d17501be6bd60226fa41b448c_0627ca68_cab_497afcaf\triagedump.dmp

These files may be available here:

C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_redis-server.exe_6be21d121b71175d17501be6bd60226fa41b448c_0627ca68_cab_497afcaf

Analysis symbol:
Rechecking for solution: 0
Report Id: 71fd018c-05e5-11e5-80c1-000d3a20bbd9
Report Status: 0
Hashed bucket:

—
Reply to this email directly or view it on GitHub
#244 (comment).

[nmehlei]

Honestly, I'm getting there, but I would rather use windows :/

[bruceliu2008]

Then you have to take risks on your own. Because as you can see, nobody
will support you.
On 29 May 2015 18:04, "nmehlei" notifications@github.com wrote:

Honestly, I'm getting there, but I would rather use windows :/

—
Reply to this email directly or view it on GitHub
#244 (comment).

[orangemocha]

As the full dump is rather large (3 GiB), how should I send you the link?

@nmehlei : can you email the link to email edited

[nmehlei]

@orangemocha
Yep, you have mail

[orangemocha]

@nmehlei , got it!

[orangemocha]

This is fixed in version 2.8.21. https://github.com/MSOpenTech/redis/releases/tag/win-2.8.21