Evenstore kept in unhealthy state after SIGABRT due to Out of Memory #1481

t-hg · 2017-11-06T09:19:02Z

We are currently running the EventStore in a docker environment (https://github.com/EventStore/eventstore-docker). As the docker container run out of memory we got the following log output:

ApplicationName='df', CommandLine='-P /data/db', CurrentDirectory='', Native error= Out of memory
[00001,37,05:21:13.575,ERROR] ---!!! VERY SLOW QUEUE MSG [MonitoringQueue]: GetFreshStats - 28469ms. Q: 0/0.
[00001,11,05:21:27.258,INFO] ES TcpConnection closed [05:21:27.258: N172.18.0.2:55338, L172.18.0.20:1113, {e4723756-1079-49c8-9827-c427e92b21c7}]:Received bytes: 12086688, Sent bytes: 137165014
[00001,11,05:21:31.077,INFO] ES TcpConnection closed [05:21:31.077: N172.18.0.2:55338, L172.18.0.20:1113, {e4723756-1079-49c8-9827-c427e92b21c7}]:Send calls: 179799, callbacks: 179799
[00001,11,05:21:31.629,INFO] ES TcpConnection closed [05:21:31.629: N172.18.0.2:55338, L172.18.0.20:1113, {e4723756-1079-49c8-9827-c427e92b21c7}]:Receive calls: 166143, callbacks: 166142
[00001,11,05:21:31.629,INFO] ES TcpConnection closed [05:21:31.629: N172.18.0.2:55338, L172.18.0.20:1113, {e4723756-1079-49c8-9827-c427e92b21c7}]:Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 166138
STATE CUE CARD: (? means a positive number, usually 1 or 2, * means any number)
	0x0	- starting (GOOD, unless the thread is running managed code)
	0x1	- running (BAD, unless it's the gc thread)
	0x2	- detached (GOOD, unless the thread is running managed code)
	0x?03	- async suspended (GOOD)
	0x?04	- self suspended (GOOD)
	0x?05	- async suspend requested (BAD)
	0x?06	- self suspend requested (BAD)
	0x*07	- blocking (GOOD)
	0x?08	- blocking with pending suspend (GOOD)
--thread 0x7ff56c0634f0 id 0x7ff4c7554700 [(nil)] state 1  
--thread 0x7ff5701ce030 id 0x7ff4c7755700 [(nil)] state 1  
--thread 0x7ff554006030 id 0x7ff4c7956700 [(nil)] state 1  
--thread 0x7ff54c085a00 id 0x7ff4c7b57700 [(nil)] state 1  
--thread 0x7ff5500b80d0 id 0x7ff4e15d5700 [(nil)] state 1  
--thread 0x7ff544002570 id 0x7ff4e17d6700 [(nil)] state 1  
--thread 0x7ff5480b0550 id 0x7ff4e19d7700 [(nil)] state 1  
--thread 0x7ff53c0374b0 id 0x7ff4e1bd8700 [(nil)] state 1  
--thread 0x7ff5400b19e0 id 0x7ff4e1dd9700 [(nil)] state 1  
--thread 0x7ff5340bb5c0 id 0x7ff4e1fda700 [(nil)] state 1  
--thread 0x7ff538059b20 id 0x7ff4e21db700 [(nil)] state 1  
--thread 0x7ff52c02b370 id 0x7ff4f08ea700 [(nil)] state 1  
--thread 0x7ff530001e40 id 0x7ff4f0aeb700 [(nil)] state 1  
--thread 0x7ff5240030a0 id 0x7ff4f0cec700 [(nil)] state 1  
--thread 0x7ff528003ba0 id 0x7ff4f0eed700 [(nil)] state 1  
--thread 0x7ff51c002ee0 id 0x7ff4f10ee700 [(nil)] state 1  
--thread 0x7ff520002da0 id 0x7ff4f12ef700 [(nil)] state 1  
--thread 0x7ff514003300 id 0x7ff4f14f0700 [(nil)] state 1  
--thread 0x7ff518003460 id 0x7ff4f16f1700 [(nil)] state 1  
--thread 0x7ff50c002f20 id 0x7ff4f18f2700 [(nil)] state 1  
--thread 0x7ff510003150 id 0x7ff4f1af3700 [(nil)] state 1  
--thread 0x7ff5040035b0 id 0x7ff4f4d42700 [(nil)] state 1  
--thread 0x7ff4fc002d50 id 0x7ff4f5144700 [(nil)] state 1  
--thread 0x4fa5490 id 0x7ff4f5345700 [(nil)] state 1  
--thread 0x7ff58c00ba70 id 0x7ff4f8a68700 [(nil)] state 1  
--thread 0x7ff588004fb0 id 0x7ff4f8e6a700 [(nil)] state 1  
--thread 0x7ff57c0019f0 id 0x7ff4f906b700 [(nil)] state 1  
--thread 0x7ff5800bf220 id 0x7ff4fb9fc700 [(nil)] state 1  
--thread 0x7ff574022740 id 0x7ff4fbbfd700 [(nil)] state 1  
--thread 0x7ff5780d35d0 id 0x7ff4fbdfe700 [(nil)] state 1  
--thread 0x7ff57001ff30 id 0x7ff50087e700 [(nil)] state 1  
--thread 0x7ff5540050f0 id 0x7ff500a7f700 [(nil)] state 1  
--thread 0x7ff54c0b7210 id 0x7ff500c80700 [(nil)] state 1  
--thread 0x7ff544004200 id 0x7ff501082700 [(nil)] state 1  
--thread 0x7ff5480a3cf0 id 0x7ff501283700 [(nil)] state 1  
--thread 0x7ff53c040890 id 0x7ff501484700 [(nil)] state 1  
--thread 0x7ff54009a170 id 0x7ff501685700 [(nil)] state 1  
--thread 0x7ff5380516b0 id 0x7ff501a87700 [(nil)] state 1  
--thread 0x7ff52c010e80 id 0x7ff502233700 [(nil)] state 1  
--thread 0x7ff53000a100 id 0x7ff502851700 [(nil)] state 1  
--thread 0x7ff524001d10 id 0x7ff502a52700 [(nil)] state 1  
--thread 0x7ff528002c60 id 0x7ff502c53700 [(nil)] state 1  
--thread 0x7ff51c001fa0 id 0x7ff5035fd700 [(nil)] state 1  
--thread 0x7ff520001c40 id 0x7ff5037fe700 [(nil)] state 1  
--thread 0x7ff514001a50 id 0x7ff5039ff700 [(nil)] state 1  
--thread 0x7ff518001a50 id 0x7ff558774700 [(nil)] state 1  
--thread 0x7ff50c001c40 id 0x7ff558975700 [(nil)] state 1  
--thread 0x7ff510001e80 id 0x7ff558b76700 [(nil)] state 1  
--thread 0x7ff504001d10 id 0x7ff558d77700 [(nil)] state 1  
--thread 0x7ff508003280 id 0x7ff558f78700 [(nil)] state 1  
--thread 0x7ff4fc0008e0 id 0x7ff55957b700 [(nil)] state 1  
--thread 0x7ff5080008e0 id 0x7ff55a5f6700 [(nil)] state 1  
--thread 0x7ff5040008e0 id 0x7ff55a7f7700 [(nil)] state 1  
--thread 0x7ff5100008e0 id 0x7ff55a9f8700 [(nil)] state 1  
--thread 0x7ff5180008e0 id 0x7ff55adfa700 [(nil)] state 1  
--thread 0x7ff5140008e0 id 0x7ff55affb700 [(nil)] state 1  
--thread 0x7ff5200008e0 id 0x7ff55b1fc700 [(nil)] state 1  
--thread 0x7ff51c0008e0 id 0x7ff55b3fd700 [(nil)] state 1  
--thread 0x7ff5280008e0 id 0x7ff55b5fe700 [(nil)] state 1  
--thread 0x7ff5240008e0 id 0x7ff55b7ff700 [(nil)] state 1  
--thread 0x7ff5300008e0 id 0x7ff5906ff700 [(nil)] state 1  
--thread 0x7ff52c0008e0 id 0x7ff590aff700 [(nil)] state 1  
--thread 0x7ff5380008e0 id 0x7ff590e6b700 [(nil)] state 1  
--thread 0x7ff5340008e0 id 0x7ff59106c700 [(nil)] state 1  
--thread 0x7ff5400008e0 id 0x7ff59126d700 [(nil)] state 1  
--thread 0x7ff53c0008e0 id 0x7ff59146e700 [(nil)] state 1  
--thread 0x7ff5480008e0 id 0x7ff59166f700 [(nil)] state 1  
--thread 0x7ff5500008e0 id 0x7ff5919fc700 [(nil)] state 1  
--thread 0x7ff54c0008e0 id 0x7ff591bfd700 [(nil)] state 1  
--thread 0x7ff5540008e0 id 0x7ff591dfe700 [(nil)] state 1  GC INITIATOR
--thread 0x7ff5700008e0 id 0x7ff591fff700 [(nil)] state 1  
--thread 0x7ff5780008e0 id 0x7ff5925fb700 [(nil)] state 1  
--thread 0x7ff5740008e0 id 0x7ff5927fc700 [(nil)] state 1  
--thread 0x7ff5800008e0 id 0x7ff5929fd700 [(nil)] state 1  
--thread 0x7ff5880008e0 id 0x7ff592bfe700 [(nil)] state 1  
--thread 0x7ff5840008e0 id 0x7ff592dff700 [(nil)] state 1  
--thread 0x7ff5440008e0 id 0x7ff594c69700 [(nil)] state 1  
--thread 0x7ff58c0008e0 id 0x7ff594e6a700 [(nil)] state 1  
--thread 0x7ff574030160 id 0x7ff5966da700 [(nil)] state 1  
--thread 0x44dff60 id 0x7ff5967d8740 [(nil)] state 1  
WAITING for 1 threads, got 0 suspended
suspend_thread suspend took 216 ms, which is more than the allowed 200 ms
Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Net.Sockets.Socket.cancel_blocking_socket_operation (System.Threading.Thread) <0x0005a>
  at System.Net.Sockets.SafeSocketHandle.ReleaseHandle () <0x0027b>
  at System.Runtime.InteropServices.SafeHandle.DangerousReleaseInternal (bool) <0x00194>
  at System.Runtime.InteropServices.SafeHandle.InternalDispose () <0x00027>
  at System.Runtime.InteropServices.SafeHandle.Dispose (bool) <0x00023>
  at System.Runtime.InteropServices.SafeHandle.Dispose () <0x00015>
  at System.Net.Sockets.Socket.Dispose (bool) <0x00073>
  at System.Net.Sockets.Socket.Dispose () <0x00015>
  at System.Net.Sockets.Socket.Close (int) <0x0001f>
  at EventStore.Transport.Tcp.TcpConnection.<CloseInternal>m__1 () <0x0001f>
  at EventStore.Transport.Tcp.Helper.EatException (System.Action) <0x00015>
  at EventStore.Transport.Tcp.TcpConnection.CloseInternal (System.Net.Sockets.SocketError,string) <0x00a3f>
  at EventStore.Transport.Tcp.TcpConnection.Close (string) <0x00043>
  at EventStore.Core.Services.Transport.Tcp.TcpConnectionManager.Stop (string) <0x0024e>
  at EventStore.Core.Services.Transport.Tcp.TcpConnectionManager.Handle (EventStore.Core.Messages.TcpMessage/HeartbeatTimeout) <0x0008b>
  at EventStore.Core.Services.Transport.Tcp.TcpConnectionManager/SendToWeakThisEnvelope.ReplyWith<T_REF> (T_REF) <0x00070>
  at EventStore.Core.Services.TimerService.TimerMessage/Schedule/<Create>c__AnonStorey0`1<T_REF>.<>m__0 () <0x00051>
  at EventStore.Core.Services.TimerService.TimerMessage/Schedule.Reply () <0x00019>
  at EventStore.Core.Services.TimerService.TimerService.OnTimerCallback (EventStore.Core.Services.TimerService.IScheduler,object) <0x00047>
  at EventStore.Core.Services.TimerService.ThreadBasedScheduler.DoTiming () <0x001a3>
  at System.Threading.ThreadHelper.ThreadStart_Context (object) <0x00083>
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) <0x001f5>
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) <0x00023>
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object) <0x0005b>
  at System.Threading.ThreadHelper.ThreadStart () <0x0002f>
  at (wrapper runtime-invoke) object.runtime_invoke_void__this__ (object,intptr,intptr,intptr) <0x000c8>

Native stacktrace:

	eventstored() [0x4432dd]
	/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7ff595c9c390]
	/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38) [0x7ff5956e0428]
	/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a) [0x7ff5956e202a]
	eventstored() [0x5a0739]
	eventstored() [0x5a0947]
	eventstored() [0x5a09f2]
	eventstored() [0x596d0c]
	eventstored() [0x597c5d]
	[0x40d5193b]

Debug info from gdb:


=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

[00001,70,05:21:35.382,INFO] External TCP connection accepted: [Normal, 172.18.0.4:51576, L172.18.0.20:1113, {425871ec-b813-48a4-bec4-27ccf8858161}].
[00001,74,05:21:35.476,INFO] Connection 'external-normal' ({425871ec-b813-48a4-bec4-27ccf8858161}) identified by client. Client connection name: 'ESJC-9678c004-d930-49f0-bf51-35c4048dfed9', Client version: V2.
[00001,11,05:21:37.459,INFO] Connection 'external-normal:ESJC-0b3f085e-3ada-4a58-af62-83781f2c36ef' [172.18.0.2:55338, {e4723756-1079-49c8-9827-c427e92b21c7}] closed: Success.
[00001,24,05:21:37.459,INFO] ES TcpConnection closed [05:21:37.459: N172.18.0.4:51576, L172.18.0.20:1113, {425871ec-b813-48a4-bec4-27ccf8858161}]:Received bytes: 141, Sent bytes: 44
[00001,24,05:21:37.476,INFO] ES TcpConnection closed [05:21:37.476: N172.18.0.4:51576, L172.18.0.20:1113, {425871ec-b813-48a4-bec4-27ccf8858161}]:Send calls: 2, callbacks: 2
[00001,24,05:21:37.476,INFO] ES TcpConnection closed [05:21:37.476: N172.18.0.4:51576, L172.18.0.20:1113, {425871ec-b813-48a4-bec4-27ccf8858161}]:Receive calls: 4, callbacks: 4
[00001,24,05:21:37.476,INFO] ES TcpConnection closed [05:21:37.477: N172.18.0.4:51576, L172.18.0.20:1113, {425871ec-b813-48a4-bec4-27ccf8858161}]:Close reason: [Success] Socket closed
[00001,41,05:21:37.479,DEBUG] Lost connection from 172.18.0.2:55338
[00001,24,05:21:37.491,INFO] Connection 'external-normal:ESJC-9678c004-d930-49f0-bf51-35c4048dfed9' [172.18.0.4:51576, {425871ec-b813-48a4-bec4-27ccf8858161}] closed: Success.
[00001,43,05:21:37.491,DEBUG] Lost connection from 172.18.0.4:51576
[00001,54,05:21:37.309,INFO] External TCP connection accepted: [Normal, 172.18.0.2:37928, L172.18.0.20:1113, {1310f4e7-4d43-4f49-ad5c-5a71eb755857}].
[00001,85,05:21:41.275,INFO] Connection 'external-normal' ({1310f4e7-4d43-4f49-ad5c-5a71eb755857}) identified by client. Client connection name: 'ESJC-0b3f085e-3ada-4a58-af62-83781f2c36ef', Client version: V2.
[00001,23,05:21:42.945,DEBUG] Could not get free mem on linux, received memory info raw string: []
ApplicationName='free', CommandLine='-b', CurrentDirectory='', Native error= Out of memory
[00001,23,05:21:42.956,DEBUG] Could not get drive name for directory '/data/db' on Unix.
ApplicationName='df', CommandLine='-P /data/db', CurrentDirectory='', Native error= Out of memory

This seemed to have happened while EventStore was trying to gather information from the operating system for monitoring and stats.

Now the issue we faced, is that the EventStore kept running in an unhealthy state after this exception, instead of terminating all processes and letting docker to stop or even restart the container.

Probably there needs to be a better exception handling for exceptions returned by the ShellExecutor.cs. Currently, if I see it right, all exceptions are swallowed (e.g. in EsDriveInfo.cs).

Any help would be very appreciated.

The text was updated successfully, but these errors were encountered:

drozzy · 2018-09-03T02:01:48Z

I have the same issue. Did you ever get any feedback on this?

t-hg · 2018-09-03T04:38:27Z

Nope

shaan1337 · 2018-09-03T04:46:56Z

@drozzy @uglywizkid we're very sorry for the delay. Based on some recent feedback we've identified with the help of one of our customers that mono 4.6 had some GC hanging issues when running low on memory. with tests on mono 4.8, the GC hanging issues were gone. we're looking at moving to mono 5 in the near future, this should in principle fix the above issues.

drozzy · 2018-09-03T16:40:46Z

So... which version is the docker file using @shaan1337 ? And is there any way I can switch it to 4.8?

https://hub.docker.com/r/eventstore/eventstore/~/dockerfile/

Thanks

drozzy · 2018-09-03T16:45:15Z

It seems like 16.04 Ubuntu is using Mono 4.2 (am I right?)... which is what Docker image is based on:
https://packages.ubuntu.com/xenial/mono-complete

shaan1337 · 2018-09-03T18:17:50Z

@drozzy The binary in our ubuntu package (eventstored) comes bundled with the mono runtime (built using mkbundle), so there isn't an easy way to change it (afaik). We currently use mono 4.6.2. You can download binaries without the mono runtime from our website: https://eventstore.org/downloads/. This will however require updating the Dockerfile and eventstore will not run as a daemon.

Here are the steps:

Create a file /etc/apt/sources.list.d/mono-official.list:
deb http://download.mono-project.com/repo/ubuntu wheezy/snapshots/4.8.1 main
sudo apt-get update && sudo apt-get install mono-complete
Download https://eventstore.org/downloads/EventStore-OSS-Mono-Ubuntu-16.04-v4.1.1-hotfix1.tar.gz, Extract and run mono EventStore.ClusterNode.exe --help

Please note that we also haven't heavily tested EventStore on mono 4.8 yet so it may be risky (we're planning to move directly to 5.x)

shaan1337 · 2018-09-03T18:21:35Z

@drozzy Increasing the amount of memory allocated to the container should resolve these issues though. You can also decrease number of cached chunks (2 by default = 512 MB), please see options: --cached-chunks and --chunks-cache-size

drozzy · 2018-09-03T20:12:30Z

Thanks @shaan1337 , I'm looking at command line options, and I don't see the default of 2 * 512MB there:
https://eventstore.org/docs/server/command-line-arguments/index.html

The default CACHED_CHUNKS value is -1 (Not sure what that means)
and default CHUNKS_CACHE_SIZE is 536871424 (I assume bytes, which is roughly 537 MB)

(I filed an issue with docs to improve the unit descriptions: https://github.com/EventStore/docs.geteventstore.com/issues/277)

drozzy · 2018-09-04T04:01:48Z

Setting CACHED_CHUNKS=0 did nothing, I'm still getting the same error message... :(

Please advise.

gregoryyoung · 2018-09-04T13:49:02Z

How did you set it?

…

On Tue, Sep 4, 2018 at 5:01 AM Andriy Drozdyuk ***@***.***> wrote: Setting CACHED_CHUNKS=0 did *nothing*, I'm still getting the same error message... :( Please advise. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1481 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAXRWpcYno-Zl8uRfwj8Jon1Gtyg9Sq4ks5uXfsvgaJpZM4QS74G> .

-- Studying for the Turing test

drozzy · 2018-09-04T20:56:27Z

With env variable as :

EVENSTORE_CACHED_CHUNKS=0

Inside my docker swarm stack yml file.

gregoryyoung · 2018-09-05T10:08:23Z

When starting up EventStore prints out its configuration (and where it got varying things from). Is it printing your configuration option as being set?

drozzy · 2018-09-05T21:19:22Z

Looks like it.

Maybe it has something to do with CHUNK INITIAL READER COUNT?

Here is partial output:

[00001,01,20:49:22.711]
ES VERSION:               4.1.1.0 (HEAD/cd0eafec71baffbcc0b9a4bb7c58e996cf290d34, Wed, 16 May 2018 11:08:07 +0400)
[00001,01,20:49:22.737] OS:                       Linux (Unix 4.4.0.131)
[00001,01,20:49:22.756] RUNTIME:                  4.6.2 (Stable 4.6.2.16/ac9e222) (64-bit)
[00001,01,20:49:22.756] GC:                       2 GENERATIONS
[00001,01,20:49:22.756] LOGS:                     /var/log/eventstore
[00001,01,20:49:22.769] MODIFIED OPTIONS:

        START STANDARD PROJECTIONS: True (Environment Variable)
        CLUSTER GOSSIP PORT:      2112 (Environment Variable)
        CACHED CHUNKS:            0 (Environment Variable)
        RUN PROJECTIONS:          All (Environment Variable)
        INT IP:                   0.0.0.0 (Config File)
        EXT IP:                   0.0.0.0 (Config File)
        INT HTTP PREFIXES:        http://*:2112/ (Config File)
        EXT HTTP PREFIXES:        http://*:2113/ (Config File)
        ADD INTERFACE PREFIXES:   false (Config File)


DEFAULT OPTIONS:

        CONFIG:                   /etc/eventstore/eventstore.conf (<DEFAULT>)
        HELP:                     False (<DEFAULT>)
        VERSION:                  False (<DEFAULT>)
        LOG:                      /var/log/eventstore (<DEFAULT>)
        DEFINES:                  <empty> (<DEFAULT>)
        ... 
       CHUNKS CACHE SIZE:        536871424 (<DEFAULT>)

       CHUNK INITIAL READER COUNT: 5 (<DEFAULT>)
       PROJECTION THREADS:       3 (<DEFAULT>)
        WORKER THREADS:           5 (<DEFAULT>)

drozzy · 2018-09-05T21:21:50Z

First out of memory message is:

[00001,22,22:51:05.308] Could not get free mem on linux, received memory info raw string: []
ApplicationName='free', CommandLine='-b', CurrentDirectory='', Native error= Out of memory

Looks like full 2 hours after the start.

drozzy · 2018-09-05T21:23:32Z

Here is my Docker stack file (the $ variables get replaced by some values during build time). As you can see I'm just running the eventstore/eventstore:release-4.1.1-hotfix1 docker image.

version: "3.6"

services:

  eventstore:
    image: eventstore/eventstore:release-4.1.1-hotfix1
    volumes:
      - $eventstorelogpath:/var/log/eventstore
      - $eventstoredbpath:/var/lib/eventstore
    deploy:
      placement:
        constraints: [node.hostname == $eventstorenodename]
    environment: 
      - EVENTSTORE_RUN_PROJECTIONS=All
      - EVENTSTORE_START_STANDARD_PROJECTIONS=True
    networks:
      - domain

jageall · 2019-10-03T10:15:17Z

@drozzy EventStore V5 is now on a new version of mono, is this still an issue?

drozzy · 2019-10-04T06:11:17Z

Sorry guys I haven't been on ES project for a while now.

drozzy mentioned this issue Sep 3, 2018

Out of memory EventStore/eventstore-docker#55

Closed

shaan1337 added kind/bug Issues which are a software defect platform/Mono impact/medium Medium prospect of negative impact labels Mar 13, 2019

jageall added the action/close if no further feedback If no further feedback is given after a potential fix within 3 weeks the issue will be closed label Oct 3, 2019

jageall closed this as completed Oct 4, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Evenstore kept in unhealthy state after SIGABRT due to Out of Memory #1481

Evenstore kept in unhealthy state after SIGABRT due to Out of Memory #1481

t-hg commented Nov 6, 2017

drozzy commented Sep 3, 2018

t-hg commented Sep 3, 2018

shaan1337 commented Sep 3, 2018

drozzy commented Sep 3, 2018

drozzy commented Sep 3, 2018

shaan1337 commented Sep 3, 2018

shaan1337 commented Sep 3, 2018

drozzy commented Sep 3, 2018 •

edited

drozzy commented Sep 4, 2018

gregoryyoung commented Sep 4, 2018 via email

drozzy commented Sep 4, 2018

gregoryyoung commented Sep 5, 2018

drozzy commented Sep 5, 2018

drozzy commented Sep 5, 2018

drozzy commented Sep 5, 2018

jageall commented Oct 3, 2019

drozzy commented Oct 4, 2019

Evenstore kept in unhealthy state after SIGABRT due to Out of Memory #1481

Evenstore kept in unhealthy state after SIGABRT due to Out of Memory #1481

Comments

t-hg commented Nov 6, 2017

drozzy commented Sep 3, 2018

t-hg commented Sep 3, 2018

shaan1337 commented Sep 3, 2018

drozzy commented Sep 3, 2018

drozzy commented Sep 3, 2018

shaan1337 commented Sep 3, 2018

shaan1337 commented Sep 3, 2018

drozzy commented Sep 3, 2018 • edited

drozzy commented Sep 4, 2018

gregoryyoung commented Sep 4, 2018 via email

drozzy commented Sep 4, 2018

gregoryyoung commented Sep 5, 2018

drozzy commented Sep 5, 2018

drozzy commented Sep 5, 2018

drozzy commented Sep 5, 2018

jageall commented Oct 3, 2019

drozzy commented Oct 4, 2019

drozzy commented Sep 3, 2018 •

edited