[2024-01-26 16:52:49,862 INFO][105146][worrisome-fortunate-talitha][MainThread] - core.schains.monitor.main:167 - Monitor iteration completed, sleeping for 20
[2024-01-26 16:52:52,708 ERROR][113202][skale-innocent-nasty][MainThread] - core.schains.monitor.main:266 - Monitor iteration failed
Traceback (most recent call last):
  File "/usr/src/admin/core/schains/monitor/main.py", line 251, in run_monitor_for_schain
    create_and_execute_tasks(
  File "/usr/src/admin/core/schains/monitor/main.py", line 186, in create_and_execute_tasks
    is_rotation_active = skale.node_rotation.is_rotation_active(name)
  File "/usr/local/lib/python3.9/site-packages/skale/contracts/manager/node_rotation.py", line 102, in is_rotation_active
    finish_ts_reached = self.is_finish_ts_reached(schain_name)
  File "/usr/local/lib/python3.9/site-packages/skale/contracts/manager/node_rotation.py", line 106, in is_finish_ts_reached
    rotation = self.skale.node_rotation.get_rotation_obj(schain_name)
  File "/usr/local/lib/python3.9/site-packages/skale/contracts/manager/node_rotation.py", line 55, in get_rotation_obj
    rotation_data = self.contract.functions.getRotation(schain_id).call()
  File "/usr/local/lib/python3.9/site-packages/web3/contract/contract.py", line 461, in call
    return call_contract_function(
  File "/usr/local/lib/python3.9/site-packages/web3/contract/utils.py", line 96, in call_contract_function
    return_data = w3.eth.call(
  File "/usr/local/lib/python3.9/site-packages/web3/eth/eth.py", line 255, in call
    return self._durin_call(transaction, block_identifier, state_override)
  File "/usr/local/lib/python3.9/site-packages/web3/eth/eth.py", line 274, in _durin_call
    return self._call(transaction, block_identifier, state_override)
  File "/usr/local/lib/python3.9/site-packages/web3/module.py", line 68, in caller
    result = w3.manager.request_blocking(
  File "/usr/local/lib/python3.9/site-packages/web3/manager.py", line 231, in request_blocking
    response = self._make_request(method, params)
  File "/usr/local/lib/python3.9/site-packages/web3/manager.py", line 175, in _make_request
    return request_func(method, params)
  File "/usr/local/lib/python3.9/site-packages/web3/middleware/attrdict.py", line 38, in middleware
    response = make_request(method, params)
  File "/usr/local/lib/python3.9/site-packages/skale/utils/web3_utils.py", line 112, in middleware
    raise EthClientOutdatedError(outdated_client_time_msg(
skale.utils.web3_utils.EthClientOutdatedError: eth_call failed; current_time: 1706287972.7088037, latest_block_timestamp: 1706228208, allowed_ts_diff: 300
[2024-01-26 16:52:52,709 INFO][113202][skale-innocent-nasty][MainThread] - core.schains.monitor.main:167 - Monitor iteration completed, sleeping for 30
[2024-01-26 16:52:55,887 ERROR][105147][big-majestic-oval-SKALE][MainThread] - core.schains.monitor.main:266 - Monitor iteration failed
Traceback (most recent call last):
  File "/usr/src/admin/core/schains/monitor/main.py", line 251, in run_monitor_for_schain
    create_and_execute_tasks(
  File "/usr/src/admin/core/schains/monitor/main.py", line 186, in create_and_execute_tasks
    is_rotation_active = skale.node_rotation.is_rotation_active(name)
  File "/usr/local/lib/python3.9/site-packages/skale/contracts/manager/node_rotation.py", line 102, in is_rotation_active
    finish_ts_reached = self.is_finish_ts_reached(schain_name)
  File "/usr/local/lib/python3.9/site-packages/skale/contracts/manager/node_rotation.py", line 106, in is_finish_ts_reached
    rotation = self.skale.node_rotation.get_rotation_obj(schain_name)
  File "/usr/local/lib/python3.9/site-packages/skale/contracts/manager/node_rotation.py", line 55, in get_rotation_obj
    rotation_data = self.contract.functions.getRotation(schain_id).call()
  File "/usr/local/lib/python3.9/site-packages/web3/contract/contract.py", line 461, in call
    return call_contract_function(
  File "/usr/local/lib/python3.9/site-packages/web3/contract/utils.py", line 96, in call_contract_function
    return_data = w3.eth.call(
  File "/usr/local/lib/python3.9/site-packages/web3/eth/eth.py", line 255, in call
    return self._durin_call(transaction, block_identifier, state_override)
  File "/usr/local/lib/python3.9/site-packages/web3/eth/eth.py", line 274, in _durin_call
    return self._call(transaction, block_identifier, state_override)
  File "/usr/local/lib/python3.9/site-packages/web3/module.py", line 68, in caller
    result = w3.manager.request_blocking(
  File "/usr/local/lib/python3.9/site-packages/web3/manager.py", line 231, in request_blocking
    response = self._make_request(method, params)
  File "/usr/local/lib/python3.9/site-packages/web3/manager.py", line 175, in _make_request
    return request_func(method, params)
  File "/usr/local/lib/python3.9/site-packages/web3/middleware/attrdict.py", line 38, in middleware
    response = make_request(method, params)
  File "/usr/local/lib/python3.9/site-packages/skale/utils/web3_utils.py", line 112, in middleware
    raise EthClientOutdatedError(outdated_client_time_msg(
skale.utils.web3_utils.EthClientOutdatedError: eth_call failed; current_time: 1706287975.8874848, latest_block_timestamp: 1706228208, allowed_ts_diff: 300
[2024-01-26 16:52:55,888 INFO][105147][big-majestic-oval-SKALE][MainThread] - core.schains.monitor.main:167 - Monitor iteration completed, sleeping for 40
[2024-01-26 16:52:57,775 INFO][1][MainProcess][MainThread] - core.schains.cleaner:65 - Cleaner process started
[2024-01-26 16:52:57,803 INFO][125217][cleaner][MainThread] - core.schains.cleaner:110 - Cleaner procedure started.
Process cleaner:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/src/admin/core/schains/cleaner.py", line 112, in monitor
    schain_names_on_contracts = get_schain_names_from_contract(
  File "/usr/src/admin/core/schains/cleaner.py", line 136, in get_schain_names_from_contract
    schains_on_contract = skale.schains.get_schains_for_node(node_id)
  File "/usr/local/lib/python3.9/site-packages/skale/contracts/manager/schains.py", line 104, in get_schains_for_node
    schain_ids = self.schains_internal.get_schain_ids_for_node(node_id)
  File "/usr/local/lib/python3.9/site-packages/skale/contracts/manager/schains_internal.py", line 55, in get_schain_ids_for_node
    return self.contract.functions.getSchainHashesForNode(node_id).call()
  File "/usr/local/lib/python3.9/site-packages/web3/contract/contract.py", line 461, in call
    return call_contract_function(
  File "/usr/local/lib/python3.9/site-packages/web3/contract/utils.py", line 96, in call_contract_function
    return_data = w3.eth.call(
  File "/usr/local/lib/python3.9/site-packages/web3/eth/eth.py", line 255, in call
    return self._durin_call(transaction, block_identifier, state_override)
  File "/usr/local/lib/python3.9/site-packages/web3/eth/eth.py", line 274, in _durin_call
    return self._call(transaction, block_identifier, state_override)
  File "/usr/local/lib/python3.9/site-packages/web3/module.py", line 68, in caller
    result = w3.manager.request_blocking(
  File "/usr/local/lib/python3.9/site-packages/web3/manager.py", line 231, in request_blocking
    response = self._make_request(method, params)
  File "/usr/local/lib/python3.9/site-packages/web3/manager.py", line 175, in _make_request
    return request_func(method, params)
  File "/usr/local/lib/python3.9/site-packages/web3/middleware/attrdict.py", line 38, in middleware
    response = make_request(method, params)
  File "/usr/local/lib/python3.9/site-packages/skale/utils/web3_utils.py", line 112, in middleware
    raise EthClientOutdatedError(outdated_client_time_msg(
skale.utils.web3_utils.EthClientOutdatedError: eth_call failed; current_time: 1706287977.86044, latest_block_timestamp: 1706228208, allowed_ts_diff: 300
[2024-01-26 16:52:57,864 INFO][1][MainProcess][MainThread] - core.schains.cleaner:67 - Cleaner process is joined.
[2024-01-26 16:52:57,865 INFO][1][MainProcess][MainThread] - core.schains.cleaner:68 - Terminating the process
[2024-01-26 16:52:57,865 INFO][1][MainProcess][MainThread] - __main__:70 - Sleeping for 90s after run_cleaner
[2024-01-26 16:53:00,636 ERROR][113194][ill-informed-friendly-haedi][MainThread] - core.schains.monitor.main:266 - Monitor iteration failed
Traceback (most recent call last):
  File "/usr/src/admin/core/schains/monitor/main.py", line 251, in run_monitor_for_schain
    create_and_execute_tasks(
  File "/usr/src/admin/core/schains/monitor/main.py", line 186, in create_and_execute_tasks
    is_rotation_active = skale.node_rotation.is_rotation_active(name)
  File "/usr/local/lib/python3.9/site-packages/skale/contracts/manager/node_rotation.py", line 102, in is_rotation_active
    finish_ts_reached = self.is_finish_ts_reached(schain_name)
  File "/usr/local/lib/python3.9/site-packages/skale/contracts/manager/node_rotation.py", line 106, in is_finish_ts_reached
    rotation = self.skale.node_rotation.get_rotation_obj(schain_name)
  File "/usr/local/lib/python3.9/site-packages/skale/contracts/manager/node_rotation.py", line 55, in get_rotation_obj
    rotation_data = self.contract.functions.getRotation(schain_id).call()
  File "/usr/local/lib/python3.9/site-packages/web3/contract/contract.py", line 461, in call
    return call_contract_function(
  File "/usr/local/lib/python3.9/site-packages/web3/contract/utils.py", line 96, in call_contract_function
    return_data = w3.eth.call(
  File "/usr/local/lib/python3.9/site-packages/web3/eth/eth.py", line 255, in call
    return self._durin_call(transaction, block_identifier, state_override)
  File "/usr/local/lib/python3.9/site-packages/web3/eth/eth.py", line 274, in _durin_call
    return self._call(transaction, block_identifier, state_override)
  File "/usr/local/lib/python3.9/site-packages/web3/module.py", line 68, in caller
    result = w3.manager.request_blocking(
  File "/usr/local/lib/python3.9/site-packages/web3/manager.py", line 231, in request_blocking
    response = self._make_request(method, params)
  File "/usr/local/lib/python3.9/site-packages/web3/manager.py", line 175, in _make_request
    return request_func(method, params)
  File "/usr/local/lib/python3.9/site-packages/web3/middleware/attrdict.py", line 38, in middleware
    response = make_request(method, params)
  File "/usr/local/lib/python3.9/site-packages/skale/utils/web3_utils.py", line 112, in middleware
    raise EthClientOutdatedError(outdated_client_time_msg(
skale.utils.web3_utils.EthClientOutdatedError: eth_call failed; current_time: 1706287980.6362667, latest_block_timestamp: 1706228208, allowed_ts_diff: 300
[2024-01-26 16:53:00,636 INFO][113194][ill-informed-friendly-haedi][MainThread] - core.schains.monitor.main:167 - Monitor iteration completed, sleeping for 22
[2024-01-26 16:53:02,949 ERROR][113164][international-villainous-zaurak][MainThread] - core.schains.monitor.main:266 - Monitor iteration failed