Bug: hydrating cache not refreshing

[bchamagne]

Describe the problem you discovered

While I was checking production node, I stumbled upon this bug:

Screenshot from 2023-07-04

Screenshot from 2023-07-05 (see bottom right, sorry I did not make a better screenshot)

As you can see on these screenshot, the state of the hydrating cache did not change in many hours. The hydrating_timer and hydrating_task are supposed to change every 10 seconds.

So there might be some scenario where the hydrating_timer is not resetted (probably the DOWN function clause)

Describe the solution you'd like

No response

Epic

No response

[redDwarf03]

relative to #1228

[bchamagne]

Process stuck have a hydrating_task defined:

iex(archethic_node@127.0.0.1)15> :sys.get_state(pid(0, 25632, 3616))
%Archethic.OracleChain.Services.HydratingCache.State{
  mfa: {Archethic.OracleChain.Services.UCOPrice.Providers.CoinMarketCap, :fetch,
   [["usd", "eur"]]},
  ttl: :infinity,
  ttl_timer: nil,
  refresh_interval: 10000,
  value: {:ok, %{"eur" => [0.0404]}},
  hydrating_task: %Task{
    mfa: {:erlang, :apply, 2},
    owner: #PID<0.25632.3616>,
    pid: #PID<0.589.3632>,
    ref: #Reference<0.1851402563.2748383233.226335>
  },
  hydrating_timer: #Reference<0.1851402563.2748317697.70803>,
  hydrating_function_timeout: 5000
}

No msg in the mailbox:

iex(archethic_node@127.0.0.1)20> Process.info(pid(0, 25632, 3616))              
[
  registered_name: Archethic.OracleChain.Services.UCOPrice.Providers.CoinMarketCapCache,
  current_function: {:gen_server, :loop, 7},
  initial_call: {:proc_lib, :init_p, 5},
  status: :waiting,
  message_queue_len: 0,
  links: [#PID<0.30046.3616>],
  dictionary: [
    "$initial_call": {Archethic.OracleChain.Services.HydratingCache, :init, 1},
    "$ancestors": [Archethic.OracleChain.Services.ProviderCacheSupervisor,
     Archethic.OracleChain.ServiceCacheSupervisor,
     Archethic.OracleChain.Supervisor, Archethic.Supervisor, #PID<0.3261.0>]
  ],
  trap_exit: false,
  error_handler: :error_handler,
  priority: :normal,
  group_leader: #PID<0.3260.0>,
  total_heap_size: 752,
  heap_size: 376,
  stack_size: 11,
  reductions: 440676,
  garbage_collection: [
    max_heap_size: %{error_logger: true, kill: true, size: 0},
    min_bin_vheap_size: 46422,
    min_heap_size: 233,
    fullsweep_after: 65535,
    minor_gcs: 852
  ],
  suspending: []
]

task process is dead:

iex(archethic_node@127.0.0.1)21> Process.alive?(pid(0, 589, 3632))
false

Which mean the task is dead but the hydratingcache was never aware of that. There is a kill_hydrating_task fallback that did not work.