DockerEngine 20.10.10: KeyError: 'total_cache'

[jensritter]

After updating from Debian buster(10) to bullseye(11) I am getting KeyError: 'total_cache'.

installed Docker : 5:20.10.103-0debian-buster
installed check_docker: 2.2.2

Exception:
"check_docker", line 541, in check_memory adjusted_usage = inspection['memory_stats']['usage'] - inspection['memory_stats']['stats']['total_cache'] KeyError: 'total_cache' UNKNOWN: Exception raised during check': KeyError('total_cache')

inspection.json.gz

My current "fix" is to skip the adjusted_usage, and only use inspection['memory_stats']['usage']

[hessijames79]

Same here :/
Seems to be related to this issue:
hashicorp/nomad#10251

[hessijames79]

Just saw there is a pull request:
#82

[blue212121]

dirty fix with limited/no testing for Docker version 20.10.12, build 20.10.12-0ubuntu4:

change 'total_cache' to 'inactive_file' in line 523, result seems to be the same

before:

   510  # Checks
   511  #############################################################################################
   512
   513  @multithread_execution()
   514  @require_running(name='memory')
   515  def check_memory(container, thresholds):
   516      if not thresholds.units in unit_adjustments:
   517          unknown("Memory units must be one of  {}".format(list(unit_adjustments.keys())))
   518          return
   519
   520      inspection = get_stats(container)
   521
   522      # Subtracting cache to match what `docker stats` does.
   523      adjusted_usage = inspection['memory_stats']['usage'] - inspection['memory_stats']['stats']['total_cache']
   524      if thresholds.units == '%':
   525          max = 100
   526          usage = int(100 * adjusted_usage / inspection['memory_stats']['limit'])
   527      else:
   528          max = inspection['memory_stats']['limit'] / unit_adjustments[thresholds.units]
   529          usage = adjusted_usage / unit_adjustments[thresholds.units]
   530
   531      evaluate_numeric_thresholds(container=container, value=usage, thresholds=thresholds, name='memory',
   532                                  short_name='mem', min=0, max=max)
   533
   534

after:

   510  # Checks
   511  #############################################################################################
   512
   513  @multithread_execution()
   514  @require_running(name='memory')
   515  def check_memory(container, thresholds):
   516      if not thresholds.units in unit_adjustments:
   517          unknown("Memory units must be one of  {}".format(list(unit_adjustments.keys())))
   518          return
   519
   520      inspection = get_stats(container)
   521
   522      # Subtracting cache to match what `docker stats` does.
   523      adjusted_usage = inspection['memory_stats']['usage'] - inspection['memory_stats']['stats']['inactive_file']
   524      if thresholds.units == '%':
   525          max = 100
   526          usage = int(100 * adjusted_usage / inspection['memory_stats']['limit'])
   527      else:
   528          max = inspection['memory_stats']['limit'] / unit_adjustments[thresholds.units]
   529          usage = adjusted_usage / unit_adjustments[thresholds.units]
   530
   531      evaluate_numeric_thresholds(container=container, value=usage, thresholds=thresholds, name='memory',
   532                                  short_name='mem', min=0, max=max)
   533
   534

[jensritter]

This looks good.

Here some test-values from my System with 5:20.10.6~3-0~debian-bullseye:

"docker stats":
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
563e5e2ba213 sonar 1.35% 3.054GiB / 13.66GiB 22.35% 8GB / 8.1GB 730MB / 7.39GB 241

'memory_stats': {
'usage': 3598192640,
'stats': {
'active_anon': 0,
'active_file': 301445120,
'anon': 2934972416,
'anon_thp': 2768240640,
'file': 621096960,
'file_dirty': 0,
'file_mapped': 41631744,
'file_writeback': 0,
'inactive_anon': 2940768256,
'inactive_file': 319369216,
'kernel_stack': 3932160,
'pgactivate': 122199,
'pgdeactivate': 1008,
'pgfault': 3038112,
'pglazyfree': 0,
'pglazyfreed': 0,
'pgmajfault': 594,
'pgrefill': 1437,
'pgscan': 39214,
'pgsteal': 27354,
'shmem': 0,
'slab': 23162272,
'slab_reclaimable': 20154376,
'slab_unreclaimable': 3007896,
'sock': 114688,
'thp_collapse_alloc': 1089,
'thp_fault_alloc': 2442,
'unevictable': 0,
'workingset_activate': 0,
'workingset_nodereclaim': 0,
'workingset_refault': 0
},
'limit': 14671679488
}