redis OOM command not allowed when used memory > 'maxmemory'.

[dinosn]

We have a few cases in all environments where Redis shows an error as described above.

The case appears on the current OVC 2.2.1 and on the previews version 2.2.0 .

An error as the one below will appear on alerta.

https://alerta.aydo.com/#/alert/7fb54510-e33a-4c97-a34c-1b29efca2889

7fb54510 uk-dc-1: Info - ErrorCondition on 57604fac-22f3-d37f-7f6d-89bd1a7c6b0d
could not report job in error to agentcontroller
ERROR:
Remote Backtrace

Traceback (most recent call last):
~ File "/opt/jumpscale7/lib/JumpScale/grid/serverbase/Daemon.py", line 231, in processRPC
result = ffunction(data)
~ File "controller.py", line 661, in notifyWorkCompleted
self.-setJob(job, osis=saveinosis)
~ File "controller.py", line 197, in -setJob
self.redis.hset("jobs:%s"%job"gid", job"guid", jobs)
~ File "/opt/jumpscale7/lib/redis/client.py", line 1853, in hset
return self.execute-command('HSET', name, key, value)
~ File "/opt/jumpscale7/lib/redis/client.py", line 565, in execute-command
return self.parse-response(connection, command-name, options)
~ File "/opt/jumpscale7/lib/redis/client.py", line 577, in parse-response
response = connection.read-response()
~ File "/opt/jumpscale7/lib/redis/connection.py", line 574, in read-response
raise response
~ ResponseError: OOM command not allowed when used memory > 'maxmemory'.

================

Client BackTrace

File "worker.py", line 276, in
worker.run()
File "worker.py", line 205, in run
self.notifyWorkCompleted(job)
File "worker.py", line 237, in notifyWorkCompleted
reportJob()
File "worker.py", line 226, in reportJob
acclient.notifyWorkCompleted(job.--dict--)
File "", line 9, in method
File "/opt/jumpscale7/lib/JumpScale/grid/serverbase/DaemonClient.py", line 282, in sendcmd
return self.sendMsgOverCMDChannel(cmd, args, sendformat, returnformat, category=category,transporttimeout=transporttimeout)
File "/opt/jumpscale7/lib/JumpScale/grid/serverbase/DaemonClient.py", line 196, in sendMsgOverCMDChannel
raise RemoteException("Cannot execute cmd:%s/%s on server:'%s:%s' error:'%s' ((ECOID:%s))" %(category,cmd,ecodict"gid",ecodict"nid",ecodict"errormessage",ecodict"guid"), ecodict)

type/level: UNKNOWN/2
ERROR IN RPC CALL notifyWorkCompleted: ResponseError: OOM command not allowed when used memory > 'maxmemory'.. (Session:{u'roles': u'node', u'storagenode', u'storagedriver', u'storagemaster', u'encrkey': u'', u'nid': 21, u'start': 1511817177, u'netinfo': {u'ip': [u'127.0.0.1', u'mac': u'00:00:00:00:00:00', u'cidr': u'8', u'name': u'lo', u'mtu': 65536}, {u'ip': u'10.16.0.63', u'mac': u'a8:1e:84:96:45:5a', u'cidr': u'24', u'name': u'eno1', u'mtu': 1500}, {u'ip': , u'mac': u'a8:1e:84:96:45:5b', u'cidr': , u'name': u'eno2', u'mtu': 1500}, {u'ip': , u'mac': u'4a:2a:41:8c:d1:aa', u'cidr': , u'name': u'ovs-system', u'mtu': 1500}, {u'ip': u'10.16.1.63', u'mac': u'ec:0d:9a:1c:03:a0', u'cidr': u'24', u'name': u'backplane1', u'mtu': 2000}, {u'ip': , u'mac': u'ec:0d:9a:1c:03:a0', u'cidr': , u'name': u'enp4s0', u'mtu': 2000}, {u'ip': , u'mac': u'ec:0d:9a:1c:03:a1', u'cidr': , u'name': u'enp4s0d1', u'mtu': 2000}, {u'ip': u'10.16.2.63', u'mac': u'80:22:c0:ff:ee:63', u'cidr': u'24', u'name': u'enp4s0f1', u'mtu': 9000}, {u'ip': , u'mac': u'2a:a2:64:07:1b:1c', u'cidr': , u'name': u'enp4s0f1d1', u'mtu': 1500}], u'gid': 888, u'passwd': u'****', u'user': u'', u'organization': u'myorg', u'id': u'888-21-0-72c1db8a-4639-41f0-b78c-a630ad45832c'})
Data:{u'job': {u'timeStop': 1511892964, u'result': u'openvstorage+tcp://10.16.2.63:26203/vm-670/cloud-init-vm-670@94a62f62-fa51-46e1-93df-d36f917b9133', u'errorreport': False, u'guid': u'6be9852e4a374a65bb12c3108b86f569', u'id': 2846, u'category': u'greenitglobe', u'timeStart': 1511892938.855335, u'log': True, u'timeCreate': 1511892938, u'state': u'OK', u'internal': False, u'gid': 888, u'jscriptid': 67, u'parent': None, u'args': {u'userdata': {}, u'type': u'Windows', u'name': u'vm-670', u'metadata': {u'admin-pass': u'6KSycg1gP', u'hostname': u'vm-670'}}, u'nid': 21, u'achost': u'64.253.35.36', u'sessionid': u'888-1-0-15bf7053-2055-44d8-942f-4c811642bb2e', u'wait': True, u'-meta': u'system', u'job', 1, u'roles': u'storagedriver', u'cmd': u'createmetaiso', u'queue': u'', u'timeout': 600, u'resultcode': 0, u'-ckey': u''}}

On the same topic we are starting to see redis reaching memory usage limits allocated to it.

Is it possibly the time to increase redis memory allocation values ?

Issues reported also on gogs at:
https://docs.greenitglobe.com/gig/org_support/issues/469

[dinosn]

The issue also has as result the following on the panel,

Issue reported at gogs:
https://docs.greenitglobe.com/gig/proj_gig_uk/issues/59

[dinosn]

More info at
https://docs.greenitglobe.com/gig/org_support/issues/473

[ashraffouda]

@grimpy @dinosn
the memory usage before this happens is high compared to the usage after that

rdb --command memory ~/Documents/dump-before.rdb --bytes 1000 

rdb --command memory ~/Documents/dump-after.rdb --bytes 1000           

this is the size of the eco:objects key
-rw-r--r--. 1 afouda afouda 17M Dec 5 09:45 before.json
-rw-r--r--. 1 afouda afouda 2.3M Dec 5 09:44 after.json

from what I see is that the things that took most of the me are eco:objects and queue:stats:min
also some thing I want to know " Was Redis crashed after OOM happened or not ?
cause I see less keys after the error and crashed my cause this problem
we have a limit of 100 m for max mem should we increase this limit

[ashraffouda]

Update found this in redis docs
https://redis.io/topics/admin

Make sure to setup some swap in your system (we suggest as much as swap as memory).
If Linux does not have swap and your Redis instance accidentally consumes too much memory, 
either Redis will crash for out of memory or the Linux kernel OOM killer will kill the Redis process.

[FastGeert]

The problem is that eco's have become much larger than before. This should be kept like this, because it makes the eco's a lot better.

Anyway, it seems that this triggers the redis to fill up, so we need to solve this.
AFAIK we put the eco's in redis for deduplication. So in stead of putting the ecos completely in redis, we should only put the hash of a calculated value based on the parameters we use for deduplication in redis, and only store the complete object in mongodb